Systems and user interfaces for dynamic and interactive investigation based on automatic clustering of related data in various data structures

ABSTRACT

In various embodiments, systems, methods, and techniques are disclosed for generating a collection of clusters of related data from a seed. Seeds may be generated based on seed generation strategies or rules. Clusters may be generated by, for example, retrieving a seed, adding the seed to a first cluster, retrieving a clustering strategy or rules, and adding related data and/or data entities to the cluster based on the clustering strategy. Various cluster scores may be generated based on attributes of data in a given cluster. Further, cluster metascores may be generated based on various cluster scores associated with a cluster. Clusters may be ranked based on cluster metascores. Various embodiments may enable an analyst to discover various insights related to data clusters, and may be applicable to various tasks including, for example, tax fraud detection, beaconing malware detection, malware user-agent detection, and/or activity trend detection, among various others.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/139,640, filed Dec. 23, 2013, titled “TREND DATA CLUSTERING,” whichis a continuation-in-part of U.S. patent application Ser. No.13/968,265, filed Aug. 15, 2013, titled “GENERATING DATA CLUSTERS WITHCUSTOMIZABLE ANALYSIS STRATEGIES,” and is a continuation-in-part of U.S.patent application Ser. No. 13/968,213, filed Aug. 15, 2013, titled“PRIORITIZING DATA CLUSTERS WITH CUSTOMIZABLE SCORING STRATEGIES,” eachof which claims benefit of U.S. Provisional Patent Application No.61/800,887, filed Mar. 15, 2013. Each of the above identifiedapplications is hereby incorporated by reference herein in its entiretyand for all purposes.

This application is also related to the following U.S. patentapplications: application Ser. No. 14/139,628, titled “TAX DATACLUSTERING,” application. Ser. No. 14/139,603, titled “MALWARE DATACLUSTERING,” and application. Ser. No. 14/139,713, titled “USER-AGENTDATA CLUSTERING.” Each of the above identified applications is herebyincorporated by reference herein in its entirety and for all purposes.

TECHNICAL FIELD

Embodiments of the present disclosure generally relate to data analysisand, more specifically, to generating data clusters of related dataentities with customizable analysis strategies.

BACKGROUND

In financial and security investigations an analyst may have to makedecisions regarding data entities within a collection of data. Forinstance, the analyst could have to decide whether an account dataentity represents a fraudulent bank account. However, an individual dataentity oftentimes includes insufficient information for the analyst tomake such decisions.

SUMMARY

The systems, methods, and devices described herein each have severalaspects, no single one of which is solely responsible for its desirableattributes. Without limiting the scope of this disclosure, severalnon-limiting features will now be discussed briefly.

In the example noted above, the analyst may make better decisions basedupon a collection of related data entities. For instance, two financialtransactions may be related by an identical account identifier or twoaccounts belonging to one customer may be related by an identicalcustomer identifier or other attribute (e.g., a shared phone number oraddress). Some currently available systems assist the analyst byidentifying data entities that are directly related to an initial dataentity. For example, the analyst could initiate an investigation with asingle suspicious data entity or “seed,” such as a fraudulent creditcard account. If the analyst examined this data entity by itself, thenthe analyst would not observe any suspicious characteristics. However,the analyst could request a list of data entities related to the seed bya shared attribute, such as a customer identifier. In doing so, theanalyst could discover an additional data entity, such as an additionalcredit card account, which relates to the original fraudulent accountbecause of a shared customer identifier. The analyst could then mark theadditional credit card account as potentially fraudulent, based upon therelationship of the shared customer identifier.

Although these systems can be very helpful in discovering related dataentities, they typically require the analyst to manually repeat the sameseries of searches for many investigations. Repeating the sameinvestigation process consumes time and resources, such that there areoftentimes more investigations than can be performed. Thus, analyststypically prioritize investigations based upon the characteristics ofthe seeds. However, there may be insignificant differences between theseeds, so the analyst may not be able to determine the correct priorityfor investigations. For instance, the analyst could have to choosebetween two potential investigations based upon separate fraudulentcredit card accounts. One investigation could reveal more potentiallyfraudulent credit card accounts than the other, and therefore could bemore important to perform. Yet, the characteristics of the two originalcredit card accounts could be similar, so the analyst would not be ableto choose the more important investigation. Without more information,prioritizing investigations is difficult and error prone.

According to various embodiments, a data analysis system is disclosed inwhich clusters of related data entities may be generated from initialdata entities, called “seeds.” A data entity may include any data,information or thing, such as a person, a place, an organization, anaccount, a computer, an activity, and event, and the like. Seeds may begenerated according to seed generation strategies, while clusters ofrelated data entities may be generated based on those seeds andaccording to cluster generation strategies. The system may furthergenerate a score, multiple scores, and/or metascores for each generatedcluster, and may optionally rank or prioritize the generated clustersbased on the generated metascores. High priority clusters may be ofgreater interest to an analyst as they may contain related data entitiesthat meet particular criteria related to the analyst's investigation. Inan embodiment, the system may enable an analyst to advantageously startan investigation with a prioritized cluster including many related dataentities rather than a single randomly selected data entity.

In one embodiment, a method for generating a cluster of related datafrom a seed is disclosed. This method may generally include retrieving aseed and adding the seed to a first cluster. This method may furtherinclude retrieving a cluster strategy referencing one or more databindings. Each data binding may specify a search protocol for retrievingdata. For each of the one or more data bindings, data parameters inputto the search protocol are identified, the search protocol is performedusing the identified data parameters, and data returned by the searchprotocol is evaluated for inclusion in the first cluster.

In a particular embodiment, the search protocol for a first one of thedata bindings uses the seed as the data parameters input to the searchprotocol. Further, the search protocol for a first one of the databindings returns data used as the identified data parameters input tothe search protocol for a second one of the data bindings. This processmay iteratively continue until all data bindings in the search strategyhave been executed for all available data obtained while growing thefirst cluster.

In a particular embodiment, this method may further include retrieving asecond cluster from a plurality of clusters, comparing data from thefirst cluster to data from the second cluster, and determining whetherto merge the first cluster and the second cluster based on thecomparison.

Other embodiments include, without limitation, a computer-readablemedium that includes instructions that enable a processing unit toimplement one or more aspects of the disclosed methods as well as asystem having a processor, memory, and application programs configuredto implement one or more aspects of the disclosed methods.

According to another embodiment, a computer system is disclosed thatcomprises: one or more computer readable storage devices configured tostore: one or more software modules including computer executableinstructions; a plurality of tax-related data items and propertiesassociated with respective tax-related data items, each of theproperties including associated property values, at least some of thetax-related data items including tax-return data items; a plurality oftax-fraud indicators; a clustering strategy; and at least one scoringcriterion; and one or more hardware computer processors in communicationwith the one or more computer readable storage devices and configured toexecute the one or more software modules in order to cause the computersystem to: designate one or more seeds by: accessing, from the one ormore computer readable storage devices, the tax-fraud indicators and atleast one tax-related data item; comparing the tax-fraud indicators tothe at least one tax-related data item and associated properties; andbased on the comparison and in response to determining the at least onetax-related data item is related to at least one tax-fraud indicator,designating the at least one tax-related data item as a seed; for eachdesignated seed, generate a cluster by: adding the seed to the cluster;adding one or more tax-return data items associated with the seed to thecluster; accessing, from the one or more computer readable storagedevices, the clustering strategy; and adding to the cluster, based onthe clustering strategy, one or more tax-related data items associatedwith the added tax-return data items; and score each generated clusterby: accessing, from the one or more computer readable storage devices,the least one scoring criterion; and generating a cluster score byassessing the generated duster based on the accessed at least onescoring criterion.

According to yet another embodiment, a computer-implemented is disclosedthat comprises: under control of one or more hardware computing devicesconfigured with specific computer executable instructions, enablingcommunication with one or more computer readable storage devicesconfigured to store: a plurality of tax-related data items andproperties associated with respective tax-related data items, each ofthe properties including associated property values; a plurality oftax-fraud indicators; a clustering strategy; and at least one scoringcriterion; designating one or more seeds by: accessing, from the one ormore computer readable storage devices, the tax-fraud indicators and atleast one tax-related data item; comparing the tax-fraud indicators tothe at least one tax-related data item and associated properties; andbased on the comparison and in response to determining the at least onetax-related data item is related to at least one tax-fraud indicator,designating the at least one tax-related data item as a seed; for eachdesignated seed, generating a cluster by: adding the seed to thecluster; accessing, from the one or more computer readable storagedevices, the clustering strategy; and adding to the cluster, based onthe clustering strategy, one or more tax-related data items associatedwith the seed; and scoring each generated cluster by: accessing, fromthe one or more computer readable storage devices, the least one scoringcriterion; and generating a cluster score by assessing the generatedcluster based on the accessed at least one scoring criterion.

According to another embodiment, a non-transitory computer-readablestorage medium is disclosed that stores computer-executable instructionsthat, when executed by a computer system, configure the computer systemto perform operations comprising: enabling communication with one ormore computer readable storage devices configured to store: a pluralityof tax-related data items and properties associated with respectivetax-related data items, each of the properties including associatedproperty values; a plurality of tax-fraud indicators; a clusteringstrategy; and at least one scoring criterion; designating one or moreseeds by: accessing, from the one or more computer readable storagedevices, the tax-fraud indicators and at least one tax-related dataitem; comparing the tax-fraud indicators to the at least one tax-relateddata item and associated properties; and based on the comparison and inresponse to determining the at least one tax-related data item isrelated to at least one tax-fraud indicator, designating the at leastone tax-related data item as a seed; for each designated seed,generating a cluster by: adding the seed to the cluster; accessing, fromthe one or more computer readable storage devices, the clusteringstrategy; and adding to the cluster, based on the clustering strategy,one or more tax-related data items associated with the seed; and scoringeach generated cluster by: accessing, from the one or more computerreadable storage devices, the least one scoring criterion; andgenerating a cluster score by assessing the generated cluster based onthe accessed at least one scoring criterion.

According to yet another embodiment, a computer system is disclosed thatcomprises: one or more computer readable storage devices configured tostore: one or more software modules including computer executableinstructions; and a plurality of beaconing malware-related data itemsand properties associated with respective malware-related data items,each of the properties including associated property values, thebeaconing malware-related data items including at least one of: dataitems associated with captured communications between an internalnetwork and an external network, users of particular computerizeddevices, internal Internet Protocol addresses, external InternetProtocol addresses, external domains, internal computerized devices,external computerized devices, data feed items, or host-based events;and one or more hardware computer processors in communication with theone or more computer readable storage devices and configured to executethe one or more software modules in order to cause the computer systemto: access, from the one or more computer readable storage devices, theplurality of beaconing malware-related data items; generate, based onthe accessed beaconing malware-related data items, a plurality ofconnection pairs, each of the connection pairs indicating communicationsbetween a particular internal source within the internal network and aparticular external destination that is not within the internal network;identify a plurality of connection pairs having a common internal sourceand a common external destination; generate a time series of connectionpairs based on the identified plurality of connection pairs; filter outnoise from the at least one time series; compute a variance in thefiltered at least one time series; and based on a determination that thevariance satisfies a particular threshold, designate a connection pairassociated with the filtered at least one time series as a seed, thedesignated connection pair including the common internal source and thecommon external source; generate a data item cluster based on thedesignated seed; and score the generated data item cluster.

According to another embodiment, a computer-implemented is disclosedthat comprises: under control of one or more hardware computing devicesconfigured with specific computer executable instructions, enablingcommunication with one or more computer readable storage devicesconfigured to store: a plurality of beaconing malware-related data itemsand properties associated with respective malware-related data items,each of the properties including associated property values, thebeaconing malware-related data items including at least one data itemassociated with captured communications between an internal network andan external network; accessing, from the one or more computer readablestorage devices, the plurality of beaconing malware-related data items;generating, based on the accessed beaconing malware-related data items,a plurality of connection pairs, each of the connection pairs indicatingcommunications between a particular internal source within the internalnetwork and a particular external destination that is not within theinternal network; identifying a plurality of connection pairs having acommon internal source and a common external destination; generating atime series of connection pairs based on the identified plurality ofconnection pairs; filtering out noise from the at least one time series;computing a variance in the filtered at least one time series; and basedon a determination that the variance satisfies a particular threshold,designating a connection pair associated with the filtered at least onetime series as a seed, the designated connection pair including thecommon internal source and the common external source; generating a dataitem cluster based on the designated seed; and scoring the generateddata item cluster.

According to yet another embodiment, a non-transitory computer-readablestorage medium is disclosed that stores computer-executable instructionsthat, when executed by a computer system, configure the computer systemto perform operations comprising: enabling communication with one ormore computer readable storage devices configured to store: a pluralityof beaconing malware-related data items and properties associated withrespective malware-related data items, each of the properties includingassociated property values, the beaconing malware-related data itemsincluding at least one data item associated with captured communicationsbetween an internal network and an external network; accessing, from theone or more computer readable storage devices, the plurality ofbeaconing malware-related data items; generating, based on the accessedbeaconing malware-related data items, a plurality of connection pairs,each of the connection pairs indicating communications between aparticular internal source within the internal network and a particularexternal destination that is not within the internal network;identifying a plurality of connection pairs having a common internalsource and a common external destination; generating a time series ofconnection pairs based on the identified plurality of connection pairs;filtering out noise from the at least one time series; computing avariance in the filtered at least one time series; and based on adetermination that the variance satisfies a particular threshold,designating a connection pair associated with the filtered at least onetime series as a seed, the designated connection pair including thecommon internal source and the common external source; generating a dataitem cluster based on the designated seed; and scoring the generateddata item cluster.

According to another embodiment, a computer system is disclosed thatcomprises: one or more computer readable storage devices configured tostore: one or more software modules including computer executableinstructions; and a plurality of user-agent-related data items andproperties associated with respective user-agent-related data items,each of the properties including associated property values, theuser-agent-related data items including at least one of: data itemsassociated with captured communications between an internal network andan external network, user-agent strings, users of particularcomputerized devices, internal Internet Protocol addresses, externalInternet Protocol addresses, external domains, internal computerizeddevices, external computerized devices, data feed items, or host-basedevents; and one or more hardware computer processors in communicationwith the one or more computer readable storage devices and configured toexecute the one or more software modules in order to cause the computersystem to: designate one or more seeds by: accessing, from the one ormore computer readable storage devices, a plurality ofuser-agent-related data items associated with the capturedcommunications; filtering the accessed data items to remove data itemsthat are unlikely to be related to malware activity; determining a firstset of filtered data items associated with a test time period and asecond set of filtered data items associated with a reference timeperiod; and identifying one or more filtered data items in the first setthat are not included among the filtered data items in the second setand designating each of the one or more identified filtered data itemsas a seed; and for each designated seed: generate a data item cluster;and score the generated data item cluster.

According to yet another embodiment, a computer-implemented is disclosedthat comprises: under control of one or more hardware computing devicesconfigured with specific computer executable instructions, enablingcommunication with one or more computer readable storage devicesconfigured to store: a plurality of user-agent-related data items andproperties associated with respective user-agent-related data items,each of the properties including associated property values; designatingone or more seeds by: accessing, from the one or more computer readablestorage devices, a plurality of user-agent-related data items associatedwith the captured communications; filtering the accessed data items toremove data items that are unlikely to be related to malware activity;determining a first set of filtered data items associated with a testtime period and a second set of filtered data items associated with areference time period; and identifying one or more filtered data item inthe first set that are not included among the filtered data items in thesecond set and designating each of the one or more identified filtereddata items as a seed; and for each designated seed: generating a dataitem cluster; and scoring the generated data item cluster.

According to another embodiment, a non-transitory computer-readablestorage medium is disclosed that stores computer-executable instructionsthat, when executed by a computer system, configure the computer systemto perform operations comprising: enabling communication with one ormore computer readable storage devices configured to store: a pluralityof user-agent-related data items and properties associated withrespective user-agent-related data items, each of the propertiesincluding associated property values; designating one or more seeds by:accessing, from the one or more computer readable storage devices, aplurality of user-agent-related data items associated with the capturedcommunications; filtering the accessed data items to remove data itemsthat are unlikely to be related to malware activity; determining a firstset of filtered data items associated with a test time period and asecond set of filtered data items associated with a reference timeperiod; and identifying one or more filtered data item in the first setthat are not included among the filtered data items in the second setand designating each of the one or more identified filtered data itemsas a seed; and for each designated seed: generating a data item cluster;and scoring the generated data item cluster.

According to yet another embodiment, a computer system is disclosed thatcomprises: one or more computer readable storage devices configured tostore: one or more software modules including computer executableinstructions; and a plurality of activity trend-related data items andproperties associated with respective trend-related data items, each ofthe properties including associated property values, the activitytrend-related data items including at least one of: host-based events,data items associated with captured host-based events, Internet Protocoladdresses, external domains, users, or computerized devices, whereinhosts comprise computerized devices in a network; and one or morehardware computer processors in communication with the one or morecomputer readable storage devices and configured to execute the one ormore software modules in order to cause the computer system to:designate one or more seeds by: accessing, from the one or more computerreadable storage devices, a plurality of activity trend-related dataitems associated with the host-based events; determining a group of theplurality of activity trend-related data items each indicating a sameparticular activity type and associated with a particular host;determining, based on the group of activity trend-related data items, astatistical deviation in the particular type of activity on theparticular host; and in response to determining that the statisticaldeviation satisfies a particular threshold, designating an activitytrend-related data item from the group as a seed; and for eachdesignated seed: generate a data item cluster; and score the generateddata item cluster.

According to another embodiment, a computer-implemented is disclosedthat comprises: under control of one or more hardware computing devicesconfigured with specific computer executable instructions, enablingcommunication with one or more computer readable storage devicesconfigured to store: a plurality of activity trend-related data itemsand properties associated with respective trend-related data items, eachof the properties including associated property values; designating oneor more seeds by: accessing, from the one or more computer readablestorage devices, a plurality of activity trend-related data itemsassociated with the host-based events; determining a group of theplurality of activity trend-related data items each indicating a sameparticular activity type and associated with a particular host;determining, based on the group of activity trend-related data items, astatistical deviation in the particular type of activity on theparticular host; and in response to determining that the statisticaldeviation satisfies a particular threshold, designating an activitytrend-related data item from the group as a seed; and for eachdesignated seed: generating a data item cluster; and scoring thegenerated data item cluster.

According to yet another embodiment, a non-transitory computer-readablestorage medium is disclosed that stores computer-executable instructionsthat, when executed by a computer system, configure the computer systemto perform operations comprising: enabling communication with one ormore computer readable storage devices configured to store: a pluralityof activity trend-related data items and properties associated withrespective trend-related data items, each of the properties includingassociated property values; designating one or more seeds by: accessing,from the one or more computer readable storage devices, a plurality ofactivity trend-related data items associated with the host-based events;determining a group of the plurality of activity trend-related dataitems each indicating a same particular activity type and associatedwith a particular host; determining, based on the group of activitytrend-related data items, a statistical deviation in the particular typeof activity on the particular host; and in response to determining thatthe statistical deviation satisfies a particular threshold, designatingan activity trend-related data item from the group as a seed; and foreach designated seed: generating a data item cluster; and scoring thegenerated data item cluster.

Advantageously, according to various embodiments, the disclosedtechniques provide a more effective starting point for an investigationof financial, security, and/or other data entities. An analyst may beable to start an investigation from a cluster of related data entitiesinstead of an individual data entity, which may reduce the amount oftime and effort required to perform the investigation. The disclosedtechniques may also, according to various embodiments, provide aprioritization of multiple clusters. For example, the analyst may alsoable to start the investigation from a high priority cluster, which mayallow the analyst to focus on the most important investigations.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention may be understood in detail, a more particular description ofvarious embodiments, briefly summarized above, may be had by referenceto the appended drawings and detailed description. It is to be noted,however, that the appended drawings illustrate only typical embodimentsof present disclosure and are therefore not to be considered limiting ofits scope, for present disclosure may admit to other equally effectiveembodiments.

FIG. 1 is a block diagram illustrating an example data analysis system,according to an embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating an example generation of clustersby the data analysis system, according to an embodiment of the presentdisclosure.

FIGS. 3A-3C illustrate an example growth of a cluster of related dataentities, according to an embodiment of the present disclosure.

FIG. 4 illustrates an example ranking of clusters by the data analysissystem, according to an embodiment of the present disclosure.

FIG. 5 illustrates an example cluster analysis user interface, accordingto an embodiment of the present disclosure.

FIG. 6 is a flowchart of an example method of generating clusters,according to an embodiment of the present disclosure.

FIG. 7 is a flowchart of an example method of scoring clusters,according to an embodiment of the present disclosure.

FIG. 8 illustrates components of an illustrative server computingsystem, according to an embodiment of the present disclosure.

FIG. 9 is a flowchart of an example generalized method of the dataanalysis system, according to various embodiments of the presentdisclosure.

Example Application of the Data Analysis System to Tax Fraud Detection

FIG. 10A is a flowchart of an example of a seed generation method of thedata analysis system as applied to tax fraud detection, according tovarious embodiments of the present disclosure.

FIG. 10B is a flowchart of an example of a clustering method of the dataanalysis system as applied to tax fraud detection, according to variousembodiments of the present disclosure.

FIG. 10C is a flowchart of example cluster scoring methods of the dataanalysis system as applied to tax fraud detection, according to variousembodiments of the present disclosure.

FIG. 10D illustrates an example growth of a cluster of related dataentities in a tax fraud detection application, according to anembodiment of the present disclosure.

FIG. 10E illustrates an example cluster analysis user interface of thedata analysis system as applied to tax fraud detection, according to anembodiment of the present disclosure.

Example Application of the Data Analysis System to Beaconing MalwareDetection

FIG. 11A is a flowchart of an example of a seed generation method of thedata analysis system as applied to beaconing malware detection,according to various embodiments of the present disclosure.

FIG. 11B is a flowchart of an example of a clustering method of the dataanalysis system as applied to beaconing malware detection, according tovarious embodiments of the present disclosure.

FIG. 11C is a flowchart of example cluster scoring methods of the dataanalysis system as applied to beaconing malware detection, according tovarious embodiments of the present disclosure.

FIG. 11D illustrates an example growth of a cluster of related dataentities in a beaconing malware detection application, according to anembodiment of the present disclosure.

FIG. 11E illustrates an example cluster analysis user interface of thedata analysis system as applied to beaconing malware detection,according to an embodiment of the present disclosure.

Example Application of the Data Analysis System to Malware User-AgentDetection

FIG. 12A is a flowchart of an example of a seed generation method of thedata analysis system as applied to malware user-agent detection,according to various embodiments of the present disclosure.

FIG. 12B is a flowchart of an example of a clustering method of the dataanalysis system as applied to malware user-agent detection, according tovarious embodiments of the present disclosure.

FIG. 12C is a flowchart of example cluster scoring methods of the dataanalysis system as applied to malware user-agent detection, according tovarious embodiments of the present disclosure.

FIG. 12D illustrates an example growth of a cluster of related dataentities in a malware user-agent detection application, according to anembodiment of the present disclosure.

FIG. 12E illustrates an example cluster analysis user interface of thedata analysis system as applied to malware user-agent detection,according to an embodiment of the present disclosure.

Example Application of the Data Analysis System to Activity TrendDetection

FIG. 13A is a flowchart of an example of a seed generation method of thedata analysis system as applied to activity trend detection, accordingto various embodiments of the present disclosure.

FIG. 13B is a flowchart of an example of a clustering method of the dataanalysis system as applied to activity trend detection, according tovarious embodiments of the present disclosure.

FIG. 13C is a flowchart of an example of a cluster scoring method of thedata analysis system as applied to activity trend detection, accordingto various embodiments of the present disclosure.

FIG. 13D illustrates an example growth of a cluster of related dataentities in an activity trend detection application, according to anembodiment of the present disclosure.

FIG. 13E illustrates an example cluster analysis user interface of thedata analysis system as applied to activity trend detection, accordingto an embodiment of the present disclosure.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS I. Terms

In order to facilitate an understanding of the systems and methodsdiscussed herein, a number of terms are defined below. The terms definedbelow, as well as other terms used herein, should be construed toinclude the provided definitions, the ordinary and customary meaning ofthe terms, and/or any other implied meaning for the respective terms.Thus, the definitions below do not limit the meaning of these terms, butonly provide exemplary definitions.

Ontology: Stored information that provides a data model for storage ofdata in one or more databases. For example, the stored data may comprisedefinitions for object types and property types for data in a database,and how objects and properties may be related.

Database: A broad term for any data structure for storing and/ororganizing data, including, but not limited to, relational databases(for example, Oracle database, mySQL database, and the like),spreadsheets, XML files, and text file, among others. The various terms“database,” “data store,” and “data source” may be used interchangeablyin the present disclosure.

Data Entity (Entity), Data Object (Object), or Data Item (Item): A datacontainer for information representing specific things in the world thathave a number of definable properties. For example, a data entity mayrepresent an entity such as a person, a place, an organization, anaccount, a computer, an activity, a market instrument, or other noun. Adata entity may represent an event that happens at a point in time orfor a duration. A data entity may represent a document or otherunstructured data source such as an e-mail message, a news report, or awritten paper or article. Each data entity may be associated with aunique identifier that uniquely identifies the data entity. The dataentity's attributes (for example, metadata about the data entity) may berepresented in one or more properties. The terms “data entity,” “dataobject,” and “data item” may be used interchangeably and/or synonymouslyin the present disclosure.

Entity (or Object or Item) Type: Type of a data entity (for example,Person, Event, or Document). Data entity types may be defined by anontology and may be modified or updated to include additional dataentity types. An data entity definition (for example, in an ontology)may include how the data entity is related to other data entities, suchas being a sub-data entity type of another data entity type (forexample, an agent may be a sub-data entity of a person data entitytype), and the properties the data entity type may have.

Properties: Also referred to as “metadata,” includes attributes of adata entity that represent individual data items. At a minimum, eachproperty of a data entity has a property type and a value or values.Properties/metadata associated with data entities may include anyinformation relevant to that object. For example, properties associatedwith a person data entity may include a name (for example, John Doe), anaddress (for example, 123 S. Orange Street), and/or a phone number (forexample, 800-0000), among other properties. In another example, metadataassociated with a computer data entity may include a list of users (forexample, user1, user 2, and the like), and/or an IP (internet protocol)address, among other properties.

Property Type: The type of data a property is, such as a string, aninteger, or a double. Property types may include complex property types,such as a series data values associated with timed ticks (for example, atime series), and the like.

Property Value: The value associated with a property, which is of thetype indicated in the property type associated with the property. Aproperty may have multiple values.

Link: A connection between two data objects, based on, for example, arelationship, an event, and/or matching properties. Links may bedirectional, such as one representing a payment from person A to B, orbidirectional.

Link Set: Set of multiple links that are shared between two or more dataobjects.

Seed: One or more data entities that may be used as a basis, or startingpoint, for generating a cluster. A seed may be generated, determined,and/or selected from one or more sets of data entities according to aseed generation strategy. For example, seeds may be generated from dataentities accessed from various databases and data sources including, forexample, databases maintained by financial institutions, governmententities, private entities, public entities, and/or publicly availabledata sources.

Cluster: A group or set of one or more related dataentities/objects/items. A cluster may be generated, determined, and/orselected from one or more sets of data entities according to a clustergeneration strategy. A cluster may further be generated, determined,and/or selected based on a seed. For example, a seed may comprise aninitial data entity of a cluster. Data entities related to the seed maybe determined and added to the cluster. Further, additional dataentities related to any clustered data entity may also be added to thecluster iteratively as indicated by a cluster generation strategy. Dataentities may be related by any common and/or similar properties,metadata, types, relationships, and/or the like.

Seed/Cluster Generation Strategy (or Rule): Seed and cluster generationstrategies/rules indicate processes, methods, and/or strategies forgenerating seeds and generating clusters, respectively. For example, aseed generation strategy may indicate that data entities having aparticular property (for example, data entities that are credit cardaccounts) are to be designated as seeds. In another example, a clustergeneration strategy may indicate that data entities having particularproperties in common with (or similar to) a seed or other data entity ina cluster are to be added to the cluster. Seed and/or cluster generationstrategies may specify particular searches and/or rule matches toperform on one or more sets of data entities. Execution of a seed and/orcluster generation strategy may produce layers of related data entities.Additionally, a seed/cluster generation strategy/rule may includemultiple strategies, sub-strategies, rules, and/or sub-rules.

II. Overview

According to various embodiments, a data analysis system is disclosed inwhich clusters of related data entities may be generated from initialdata entities, called “seeds.” Seeds and related data entities may beaccessed from various databases and data sources including, for example,databases maintained by financial institutions, government entities,private entities, public entities, and/or publicly available datasources. Such databases and data sources may include a variety ofinformation and data, such as, for example, personal information,financial information, tax-related information, computer network-relateddata, and/or computer-related activity data, among others. Further, thedatabases and data sources may include various relationships that linkand/or associate data entities with one another. Various data entitiesand relationships may be stored across different systems controlled bydifferent entities and/or institutions. According to variousembodiments, the data analysis system may bring together data frommultiple data sources in order to build clusters.

In various embodiments, the data analysis system may enable a user toefficiently perform analysis and investigations of various data entitiesand clusters of data entities. For example, the system may enable a user(also referred to herein as an “analyst”) to perform various financialand security investigations related to a seed (for example, an initialdata entity or data object). In such an investigation, the system mayenable an analyst to search and/or investigate several layers of relateddata entities. For example, a credit card account may be a seed that islinked by the system to various data entities including, for example,customer identifiers and/or phone numbers associated with the creditcard account. Further, the system may link, for example, various othercredit card accounts related to the customer identifiers, to the seedcredit card account. Accordingly, in various embodiments, the system mayautomatically determine and provide to a user or analyst various layersof data entities related to the seed credit card account. Such aninvestigation may, in an embodiment, enable the analyst to determinefraudulent activity. For example, if the seed credit card account wassuspected to be fraudulent, then the analyst may determine that theadditional credit card accounts may also be fraudulent. Further, if theseed credit card account was linked to other known fraudulent creditcard accounts, the analyst may determine that the seed credit cardaccount was likely to be fraudulent. As mentioned above, in such aninvestigation the analyst may discover relationships between theadditional credit card accounts and the seed credit card account throughseveral layers of related data entities. Such techniques, enabled byvarious embodiments of the data analysis system, may be particularlyvaluable for investigations in which relationships between data entitiesmay include several layers, and in which such relationships may beotherwise very difficult or impossible to manually identify.

In various embodiments, the data analysis system may automaticallygenerate, or determine, seeds based on a seed generation strategy (alsoreferred to as “seed generation rules”). For example, for a particularset of data entities, the data analysis system may automaticallygenerate, based on a seed generation strategy, seeds by designatingparticular data entities (or groups of data entities) as seeds. Examplesof various seed generation strategies are described below.

Further, in various embodiments, the data analysis system mayautomatically discover data entities related to a seed, and store theresulting relationships and related data entities together in a“cluster.” A cluster generation strategy (also referred to as “clustergeneration rules”) may specify particular searches to perform at eachstep of an investigation, or cluster generation, process. Such searchesmay produce layers of related data entities to add to the cluster.Further, according to an embodiment, multiple cluster may be mergedand/or collapsed into a single cluster when the multiple cluster shareone or more common data entities and/or properties. Thus, according toan embodiment, an analyst may start an investigation with the resultingcluster, rather than the seed alone. Starting with the cluster, theanalyst may form opinions regarding the related data entities, conductfurther analysis of the related data entities, and/or may query foradditional related data entities.

According to various embodiments, the data analysis system may furthergenerate various “cluster scores.” Cluster scores may include scoresbased on various characteristics and/or attributes associated with thecluster and/or the various data entities of the cluster. In variousembodiments, the data analysis system may also generate “clustermetascores” which may include, for example, an overall cluster score.Cluster metascores may, for example, be based on a combination ofcluster scores of a cluster associated with a seed.

Further, in various embodiments, for a particular set of data entities,multiple clusters may be generated by the data analysis system. Forexample, the data analysis system may generate multiple seeds accordingto a seed generation strategy, and then multiple clusters based on thoseseeds (and based on a cluster generation strategy). In such embodiments,the data analysis system may prioritize the multiple generated clustersbased upon cluster scores and/or cluster metascores. In an embodiment,the data analysis system may provide a user interface including adisplay of summaries of the clusters, including cluster scores, clustermetascores, and/or various other cluster information. Such summaries maybe displayed according to a prioritization of clusters. In variousembodiments, cluster prioritization may assist an analyst in selectingparticular clusters to investigate.

In various embodiments, the data analysis system may be used in variousdata analysis applications. Such applications may include, for example,financial fraud detection, tax fraud detection, beaconing malwaredetection, malware user-agent detection, activity trend detection,health insurance fraud detection, financial account fraud detection,detection of activity by networks of individuals, and/or criminalactivity detection, among various others. A number of examples of suchapplications are described in detail below in reference to, for example,FIGS. 3A-3C (financial fraud detection), 10A-10E (tax fraud detection),11A-11E (beaconing malware detection), 12A-12E (malware user-agentdetection), and 13A-13E (activity trend detection).

In the following description, numerous specific details are set forth toprovide a more thorough understanding of various embodiments of thepresent disclosure. However, it will be apparent to one of skill in theart that the systems and methods of the present disclosure may bepracticed without one or more of these specific details.

III. Examples of Data Entities, Properties, and Links

In various embodiments, different types of data entities may havedifferent property types. For example, a “Person” data entity may havean “Eye Color” property type and an “Event” data entity may have a“Date” property type. Each property as represented by data in a databasemay have a property type defined by an ontology used by the database.Further, data entities may be instantiated in a database in accordancewith a corresponding object definition for the particular data entity inthe ontology. For example, a specific monetary payment (for example, anentity of type “event”) of US$30.00 (for example, a property of type“currency” having a property value of “US$30.00”) taking place on Mar.27, 2009 (for example, a property of type “date” having a property valueof “Mar. 27, 2009”) may be stored in the database as an event objectwith associated currency and date properties as defined within theontology.

Data objects defined in an ontology may support property multiplicity.In particular, a data entity may be allowed to have more than oneproperty of the same property type. For example, a “Person” data objectmight have multiple “Address” properties or multiple “Name” properties.

A link represents a connection between two data entities and may bethrough any of a relationship, an event, and/or matching properties. Alink may be asymmetrical or symmetrical. For example, “Person” dataentity A may be connected to “Person” data entity B by a “Child Of”relationship (where “Person” data entity B has an asymmetric “Parent Of”relationship to “Person” data entity A), a “Kin Of” symmetricrelationship to “Person” data entity C, and an asymmetric “Member Of”relationship to “Organization” data entity X. The type of relationshipbetween two data entities may vary depending on the types of the dataentities. For example, “Person” data entity A may have an “Appears In”relationship with “Document” data entity Y or have a “Participate In”relationship with “Event” data entity E. As an example of an eventconnection, two “Person” data entities may be connected by an “AirlineFlight” data entity representing a particular airline flight if theytraveled together on that flight, or by a “Meeting” data entityrepresenting a particular meeting if they both attended that meeting. Inone embodiment, when two data entities are connected by an event, theyare also connected by relationships, in which each data entity has aspecific relationship to the event, such as, for example, an “AppearsIn” relationship.

As an example of a matching properties connection, two “Person” dataentities representing a brother and a sister may both have an “Address”property that indicates where they live. If the brother and the sisterlive in the same home, then their “Address” properties likely containsimilar, if not identical property values. In one embodiment, a linkbetween two data entity may be established based on similar or matchingproperties (for example, property types and/or property values) of thedata entity. These are just some examples of the types of connectionsthat may be represented by a link and other types of connections may berepresented; embodiments are not limited to any particular types ofconnections between data entities. For example, a document may containreferences to two different entities. For example, a document maycontain a reference to a payment (one data entity), and a person (asecond data entity). A link between these two data entities mayrepresent a connection between these two entities through theirco-occurrence within the same document.

Each data entity may have multiple links with another data entity toform a link set. For example, two “Person” data entities representing ahusband and a wife may be linked through a “Spouse Of” relationship, amatching “Address” property, and/or one or more matching “Event”properties (for example, a wedding). Each link, as represented by datain a database, may have a link type defined by the database ontologyused by the database.

In various embodiments, the data analysis system may access various dataentities and associated properties from various databases and datasources. Such databases and data sources may include a variety ofinformation and data, such as, for example, personal information (forexample, names, addresses, phone numbers, personal identifiers, and thelike), financial information (for example, financial accountinformation, transaction information, balance information, and thelike), tax-related information (for example, tax return data, and thelike), computer network-related data (for example, network trafficinformation, IP (Internet Protocol) addresses, user account information,domain information, network connection information, and the like),and/or computer-related activity data (for example, computer events,user actions, and the like), among others.

IV. Description of the Figures

Embodiments of the disclosure will now be described with reference tothe accompanying Figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive manner,simply because it is being utilized in conjunction with a detaileddescription of certain specific embodiments of the disclosure.Furthermore, embodiments of the disclosure may include several novelfeatures, no single one of which is solely responsible for its desirableattributes or which is essential to practicing the embodiments of thedisclosure herein described.

FIG. 1 is a block diagram illustrating an example data analysis system100, according to one embodiment. As shown in the embodiment of FIG. 1,the data analysis system 100 includes an application server 115 runningon a server computing system 110, a client 135 running on a clientcomputer system 130, and at least one database 140. Further, the client135, application server 115, and database 140 may communicate over anetwork 150, for example, to access cluster data sources 160.

The application server 115 may include a cluster engine 120 and aworkflow engine 125. The cluster engine 120 and a workflow engine 125may be software modules as described below in reference to FIG. 8.According to an embodiment, the cluster engine 120 is configured tobuild one or more clusters of related data entities, according to adefined cluster generation strategy. The cluster engine 120 may readdata from a variety of cluster data sources 160 to generate clustersfrom seeds (also referred to as “seed data entities”). Once created, theresulting clusters may be stored on the server computing system 110and/or on the database 140. The operations of the cluster engine 120 arediscussed in detail below in conjunction with FIGS. 2 and 3.

In an embodiment, the cluster engine 120 is configured to score theclusters, according to a defined scoring strategy. The score mayindicate the importance of analyzing the cluster. For instance, thecluster engine 120 may execute a scoring strategy that aggregates theaccount balances of credit card accounts within the cluster. Because,for example, a large aggregated total balance may indicate a largeliability for a financial institution, a cluster with such a large totalbalance may be considered to have a higher score relative to otherclusters with lower aggregated total balances (and, therefore, lowerscores). Thus, a cluster with a higher score relative to a cluster witha lower score may be considered more important to analyze.

In one embodiment, the cluster engine 120 organizes and presents theclusters according to the assigned scores. The cluster engine 120 maypresent summaries of the clusters and/or interactive representations ofthe clusters within the cluster analysis UI. For example, therepresentations may provide visual indications (e.g., graphs or othervisualizations) of the related data entities within the clusters. Thecluster engine 120 may generate a cluster analysis user interface (UI),such as a web application and/or a dynamic web page displayed within theclient 135. The cluster engine 120 may also allow an analyst to createtasks associated with the clusters. Example operations of the clusterengine 120 are discussed in detail below in conjunction with FIGS. 4 and5. In one embodiment, the cluster engine 120 generates clustersautomatically, for example, for subsequent review by analysts.

Analysts may also assign tasks to themselves or one another via aworkflow UI generated by the workflow engine, for example. The workflowengine 125 may consume scores generated by the cluster engine 120. Forexample, the workflow engine 125 may present an analyst with clustersgenerated, scored, and ordered by the cluster engine 120.

The client 135 may represent one or more software applications ormodules configured to present data and translate input, from theanalyst, into requests for data analyses by the application server 115.In one embodiment, the client 135 and the application server 115 may beembodied in the same software module and/or may be included in the samecomputing system. However, several clients 135 may execute on the clientcomputer 130, and/or several clients 135 on several client computers 130may interact with the application server 115. In one embodiment, theclient 135 may be a browser accessing a web service.

While the client 135 and application server 115 are shown running ondistinct computing systems, the client 135 and application server 115may run on the same computing system. Further, the cluster engine 120and the workflow engine 125 may run on separate applications servers115, on separate server computing systems, or some combination thereof.Additionally, a history service may store the results generated by ananalyst relative to a given cluster

In one embodiment, the cluster data sources 160 provide data availableto the duster engine to create or generate seeds and/or to create orgenerate clusters from a seed or a set of seeds. Such data sources mayinclude relational data sources, web services data, XML data, and thelike. Further, such data sources may include a variety of informationand data, for example, personal information, financial information,tax-related information, computer network-related data, and/orcomputer-related activity data, among others. For example, the datasources may be related to customer account records stored by a financialinstitution. In such a case, the data sources may include a credit cardaccount data, bank account data, customer data, and transaction data.The data may include data attributes such as account numbers, accountbalances, phone numbers, addresses, and transaction amounts, and thelike. Of course, cluster data sources 160 is included to berepresentative of a variety of data available to the server computersystem 110 over network 150, as well as locally available data sources.

The database 140 may be a Relational Database Management System (RDBMS)that stores the data as rows in relational tables. The term “database,”as used herein, may refer to an database (e.g., RDBMS or SQL database),or may refer to any other data structure, such as, for example a commaseparated values (CSV), extensible markup language (XML), text (TXT)file, flat file, spreadsheet file, and/or any other widely used orproprietary format. While the database 140 is shown as a distinctcomputing system, the database 140 may operate on the same servercomputing system 110 as the application server 115.

FIG. 2 is a block diagram illustrating an example generation of clustersby data analysis system 200, according to an embodiment. As shown, in anembodiment the cluster engine 120 (FIG. 1) interacts with a seed list210, a cluster list 250, a cluster strategy store 230, and data bindings237. The seed list 210 may include seeds 212-1, 212-2 . . . 212-S, andthe cluster list 250 may include clusters 252-1, 252-2 . . . 252-C. Thecluster engine 120 may be configured as a software application, module,or thread that generates the clusters 252-1, 252-2 . . . 252-C from theseeds 212-1, 212-2 . . . 212-S.

Seeds 212 (including one, some, or all of seeds 212-1 through 212-S) maybe generated by the cluster engine 120 according to various seedgeneration strategies/rules. Examples of seed generation are describedbelow in reference to various example applications of the data analysissystem. According to an embodiment, once generated, seeds 212 may be thestarting point for generating a cluster 252. To generate a cluster, thecluster engine 120 may retrieve a given seed 212 from the seed list 210.The seed 212 may be a data entity or group of data entities within thedatabase 140, such as a customer name, a customer social securitynumber, an account number, and/or a customer telephone number.

The cluster engine 120 may generate the cluster 252 from the seed 212.In one embodiment, the cluster engine 120 generates the cluster 252 as acollection of data entities and the relationships between the variousdata entities. As noted above, the cluster strategy may execute databindings in order to add each additional layer of data entities to thecluster. For example, the cluster engine 120 may generate the cluster252-1 from a seed credit card account. The cluster engine 120 may firstadd the credit card account to the cluster 252-1. The cluster engine 120may then add customers related to the credit card account to the cluster252-1. The cluster engine 120 may complete the cluster 252-1 by addingadditional credit card accounts related to those customers. As thecluster engine 120 generates the cluster 252-1, the cluster engine 120may store the cluster 252-1 within the cluster list 250. The cluster252-1 may be stored as a graph data structure or other appropriate datastructure.

The cluster list 250 may be a collection of tables in the database 140.In such a case, there may be a table for the data entities of eachcluster 252, such as those of example cluster 252-1 discussed above, atable for the relationships between the various data entities, a tablefor the attributes of the data entities, and a table for scores of theclusters. The cluster list 250 may include clusters 252 from multipleinvestigations. Note that the cluster engine 120 may store portions ofclusters 252 in the cluster list 250 as the cluster engine 120 generatesthe clusters 252. Persons skilled in the art will recognize that manytechnically feasible techniques exist for creating and storing datastructures that may be used to implement the systems and methods of thedata analysis system.

The cluster strategy store 230 may include cluster strategies 232-1,232-2 . . . 232-N. Each cluster strategy may include data bindingreferences 235 to one or more data bindings 237. As noted, each databinding may be used to identify data that may grow a cluster (asdetermined by the given search strategy 232). For example, the clusterengine 120 may execute a cluster strategy 232-1 to generate the cluster252-1. Specifically, the cluster engine 120 may execute the clusterstrategy 232-1 in response to selection of that cluster strategy by ananalyst. The analyst may submit a selection of one or more clusterstrategies to perform on a seed or group of seeds to the cluster engine120 through the client 135. Alternatively, the cluster engine 120 mayautomatically select one or more cluster strategies, such as based onuser preferences or rules.

According to an embodiment, each cluster strategy 232 is configured soas to perform an investigation processes for generating a cluster 252.Again, for example, the cluster strategy 232-2 may include data bindingreferences 235 to a collection of data bindings executed to add layerafter layer of data to a cluster. The investigation process may includesearches to retrieve data entities related to a seed 212 that isselected for clustering using cluster strategy 232-2. For example, thecluster strategy 232-2 may start with a possibly fraudulent credit cardaccount as the seed 212-2. The cluster strategy 232-2 may search forcustomers related to the credit card account, and then additional creditcard accounts related to those customers. A different cluster strategy232-3 may search for customers related to the credit card account, phonenumbers related to the customers, additional customers related to thephone numbers, and additional credit card accounts related to theadditional customers, for example.

In an embodiment, cluster strategies 232 include references to at leastone data binding 237 (such as data bindings 237-1 through 237-3). Thecluster engine 120 may execute a search protocol specified by the databinding 237 to retrieve data, and the data returned by a given databinding may form a layer within the cluster 252. For instance, the databinding 237 (and/or the search protocol of the data binding 237) mayretrieve sets of customers related to an account by an account ownerattribute. The data binding 237 (and/or the search protocol of the databinding 237) may retrieve the set of related data entities from a datasource. For instance, the data binding 237-1 may specify a databasequery to perform against a database. Likewise, the data binding 237-2may define a connection and/or query to a remote relational databasesystem and the data binding 237-3 may define a connection and/or queryagainst a third-party web service. Once retrieved, the cluster strategy232 may evaluate whether the returned data should be added to a clusterbeing grown from a given seed 212.

Multiple cluster strategies 232 may reference a given data binding 237.The analyst may update the data binding 237, but typically updates thedata binding 237 only if the associated data source changes. A clusterstrategy 232 may also include a given data binding 237 multiple times.For example, executing a data binding 237 using one seed 212 maygenerate additional seeds for that data binding 237 (or generate seedsfor another data binding 237). More generally, different clusterstrategies 232-1, 232-2 . . . 232-N may include different arrangementsof various data bindings 237 to generate different types of clusters252.

The cluster strategies 232 may specify that the cluster engine 120 usean attribute from the related data entities retrieved with one databinding 237, as input to a subsequent data binding 237. The clusterengine 120 may use the subsequent data binding 237 to retrieve asubsequent layer of related date entities for the cluster 252. Forinstance, a particular cluster strategy 232 may specify that the clusterengine 120 retrieve a set of credit card account data entities with afirst data binding 237-1. That cluster strategy 232 may also specifythat the cluster engine 120 then use the account number attribute fromcredit card account data entities as input to a subsequent data binding237-2. The cluster strategy 232 may also specify filters for the clusterengine 120 to apply to the attributes before performing the subsequentdata binding 237. For instance, if the first data binding 237-1 were toretrieve a set of credit card account data entities that included bothpersonal and business credit card accounts, then the cluster engine 120could filter out the business credit card accounts before performing thesubsequent data binding 237-2.

In operation, according to an embodiment, the cluster engine 120generates a cluster 252-1 from a seed 212-1 by first retrieving acluster strategy 232. Assuming the analyst selected a cluster strategy232-2, the cluster engine 120 would retrieve the cluster strategy 232-2from the cluster strategy store 230. The cluster engine 120 may thenretrieve the seed 212-1 as input to the cluster strategy 232-2. Thecluster engine 120 may execute the cluster strategy 232-2 by retrievingsets of data by executing data bindings 237 referenced by the clusterstrategy 232-2. For example, the cluster strategy 232-2 may execute databindings 237-1, 237-2, and 237-3. Accordingly, the cluster engine 120may evaluate data returned by each data binding 237 to determine whetherto use that data to grow the cluster 252-1. The cluster engine 120 maythen use elements of the returned data as input to the next data binding237. Of course, a variety of execution paths are possible for the databindings 237. For example, assume one data binding 237 returned a set ofphone numbers. In such a case, another data binding 237 may evaluateeach phone number individually. As another example, one data binding 237may use input parameters obtained by executing multiple, other databindings 237. More generally, the cluster engine 120 may retrieve datafor each data binding referenced by the cluster strategy 232-2. Thecluster engine 120 may then store the complete cluster 252-1 in thecluster list 250.

As the cluster engine 120 generates the clusters 252-1, 252-2 . . .252-C from seeds 212-1, 212-2 . . . 212-S, the cluster list 250 mayinclude overlapping clusters 252. For example, two clusters 252-1 and252-C may overlap if both clusters 252-1 and 252-C include a common dataentity. In an example, a larger cluster 252 formed by merging twosmaller clusters 252-1 and 252-C may be a better investigation startingpoint than the smaller clusters 252-1 and 252-C individually. The largercluster 252 may provide additional insight or relationships, which maynot be available if the two clusters 252-1 and 252-C remain separate.

In an embodiment, the cluster engine 120 includes a resolver 226 that isconfigured to detect and merge two or more overlapping clusters 252together. For example, the resolver 226 may compare the data entitieswithin a cluster 252-1 to the data entities within each one of the otherclusters 252-2 through 252-C. If the resolver 226 finds the same dataentity within the cluster 252-1 and a second cluster 252-C, then theresolver 226 may merge the two clusters 252-1 and 252-C into a singlelarger cluster 252. For example, the cluster 252-1 and cluster 252-C mayboth include the same customer. The resolver 226 may compare the dataentities of cluster 252-1 to the data entities of cluster 252-C anddetect the same customer in both clusters 252. Upon detecting the samecustomer in both clusters 252, the resolver 226 may merge the cluster252-1 with cluster 252-C. The resolver 226 may test each pair ofclusters 252 to identify overlapping clusters 252. Although the largerclusters 252 may be better investigation starting points, an analyst maywant to understand how the resolver 226 formed the larger clusters 252.Accordingly, the resolver 226, may store a history of each merge.

In an embodiment, cluster merging (for example, by resolver 226) may beoptionally disabled for particular types of data entities, and/orparticular data entities. For example, when a particular data entity, ortype of data entity, is so common that it may be included in manydifferent clusters (for example, an institutional entity such as abank), merging of cluster based on that common entity (for example, theparticular bank) or common type of entity (for example, banks ingeneral) may be disabled. In another embodiment, cluster may be mergedonly when they share two or more common data entities and/or otherproperties. In an embodiment, when two clusters are determined to sharea data entity that this very common (such that they cluster may not bemerged based on that entity) the system may automatically determinewhether the two clusters share one or more other data entities and/orproperties such that they may be merged. In various embodiments, clustermerging may be disabled based on other criteria. For example, clustermerging between two related clusters may be disabled when one or both ofthe two clusters reach a particular size (for example, include aparticular number of data entities).

After the cluster engine generates a group of clusters from a givencollection of seeds (and after merging or resolving the cluster), thecluster engine 120 may score, rank, and/or otherwise order the clustersrelative to a scoring strategy 442. In some embodiments, clusters arescored and provided to the analysis without resolving.

In one embodiment, the analysis system 100, and more specifically, thecluster engine 120, receives a request for cluster generation. Inresponse to the request, a list of seeds may be generated, clusters maybe generated based on those seeds, and the clusters may be ranked,ordered, and presented to analysts. In an embodiment, the cluster engine120 may consume seeds generated by other systems. Alternatively, inother embodiments, cluster engine 120 may generate the seeds 212-1,212-2 . . . 212-S. For instance, the cluster engine 120 may include aseed generation strategy (also referred to as a “lead generationstrategy”) that identifies data entities, or groups of data entities, aspotential seeds 212. The seed generation (or lead generation) strategymay apply to a particular business type, such as credit cards, stocktrading, or insurance claims, and may be run against a cluster datasource 160 or an external source of information.

In an embodiment, the analysis system 100 may not include data bindingsas described above. Rather, according to an embodiment, the analysissystem 100 may include one or more interfaces and/or connections tovarious internal and/or external data stores of data entities and/orother information. According to an embodiment, the system may include ageneric interface and/or connection to various internal and/or externaldata stores of data entities and/or other information. For example, theanalysis system 100 may include a generic data interface through whichthe system may search, access, and/or filter various data entityinformation during seed and/or cluster generation. The generic interfacemay include various aspects that enable searching, accessing, and/orfiltering of data. For example, the generic interface may access variousdata sources that each have differing data formats. The genericinterface may accordingly covert and/or filter the accessed data to acommon format. Alternatively, the data sources may include functionalitythrough which stored data may be searched and/or converted to a standardformat automatically. In an embodiment, the generic interface may enableFederated search of multiple data stores of data entity-relatedinformation. Accordingly, in various embodiments, the analysis system100 may access various data sources for data entity clustering and seedgeneration.

V. Example Cluster Generation

FIGS. 3A-3C illustrate an example growth of a cluster 252 of relateddata entities, according to an embodiment. As shown in FIG. 3A, anexample cluster 252 may include a seed entity 302, links 303-1 and303-2, and related data entities 305-1 and 305-2. The cluster 252 may bebased upon a seed 212 (for example, data entity 302). The cluster engine120 may build the cluster 252 by executing a cluster strategy 232 withthe following searches:

-   -   Find seed owner    -   Find all phone numbers related to the seed owner    -   Find all customers related to the phone numbers    -   Find all accounts related to the customers    -   Find all new customers related to the new accounts

In the example, assuming the seed 212 is fraudulent credit card account,the cluster engine 120 would add the credit card account to the cluster252 as the seed entity 302. The cluster engine 120 may then use theaccount owner attribute of the credit card account as input to a databinding 237. The cluster engine 120 may execute the search protocol ofthe data binding 237 to retrieve the customer data identifying the ownerof the fraudulent credit card account. The cluster engine 120 would thenadd the customer data to the cluster 252 as the related data entity305-1. The cluster engine 120 would also add the account owner attributeas the link 303-1 that relates the account number to the customer dataof the owner. The cluster engine 120 would execute the next search ofthe cluster strategy 232 by inputting the customer identifier attributeof the customer data into a data binding 237 to retrieve a phone data.The cluster engine 120 would then add the phone data as the related dataentity 305-2 and the customer identifier attribute as the link 303-2between the customer data and the phone data. At this point in theinvestigation process, the cluster 252 would include the seed entity302, two links 303-1 and 303-2, and two related data entities 305-1 and305-2. That is, the cluster 252 would include the fraudulent credit cardaccount, the customer data of the owner of the credit card, and thephone number of the owner. By carrying the investigation processfurther, the cluster engine 120 may reveal further related information,for example, additional customers and/or potentially fraudulent creditcard accounts.

Turning to FIG. 3B, and continuing the example, the cluster engine 120may continue executing the cluster strategy 232 by searching foradditional account data entities related to the phone number of theowner of the fraudulent credit card account. As discussed, the phonenumber may be stored as related data entity 305-2. The cluster engine120 would input the phone owner attribute of the phone number to a databinding 237. The cluster engine 120 would execute the search protocol ofdata binding 237 to retrieve the data of two additional customers, whichthe cluster engine 120 would store as related data entities 305-3 and305-4. The cluster engine 120 would add the phone owner attribute as thelinks 303-3 and 304-4 between the additional customers and the phonenumber.

Continuing the example, FIG. 3C shows the cluster 252 after the clusterengine 120 performs the last step of the example cluster strategy 232.For example, the cluster engine 120 would use the customer identifierattribute of the related data entity 305-3 and 305-4 to retrieve and addadditional account data entities as the related data entities 305-5 and305-6. The cluster engine 120 would couple the related data entities305-5 and 305-6 to the related data entities 305-3 and 305-4 with thecustomer identifier attributes stored as links 303-5 and 303-6. Thus,the cluster 252 would include six related data entities 305 related bysix links 303, in addition to the seed entity 302.

In an embodiment, the analyst may identify and determine whether theadditional data account entities, stored as related data entities 305-5and 305-6, represent fraudulent credit card accounts more efficientlythan if the analyst started an investigation with only the seed 302. Asthe foregoing example illustrates, according to various embodiments, thedata analysis system may enable an analyst to advantageously start aninvestigation with a cluster including many related data entities (suchas the example cluster 252 with the seed entity 302 and related dataentities 305) rather than a single data entity.

VI. Example Cluster Scoring/Ranking

FIG. 4 illustrates an example ranking of clusters 252 by the dataanalysis system 100 shown in FIG. 1, according to an embodiment of thepresent disclosure. As shown, an example system 400 of FIG. 4illustrates some of the same elements as shown in FIG. 1 and FIG. 2,including the cluster engine 120 in communication with the cluster list250. In addition, FIG. 4 illustrates a scoring strategy store 440 incommunication with the cluster engine 120. The scoring strategy store440 includes scoring strategies 442-1, 442-2 . . . 442-R.

In an embodiment, the cluster engine 120 executes a scoring strategy 442to score a cluster 252. For example, the cluster engine 120 may generatea cluster (for example, via a cluster strategy/data bindings) andattempt to resolve it with existing clusters. Thereafter, the clusterengine 120 may score the resulting cluster with any scoring strategiesassociated with a given cluster generation strategy. In an embodiment,the multiple scores may be generated for a given cluster. The multiplescores may be based on various aspects, metrics, or data associated withthe cluster. In one embodiment, a cluster metascore may be generatedbased on a combination or aggregation of scores associated with a givencluster. Ordering for a group of clusters, (according to a given scoringstrategy) may be performed on demand when requested by a client.Alternatively, the analyst may select a scoring strategy 442 through theclient 135 and/or the analyst may include the selection within a scriptor configuration file. In another alternative, the data analysis systemmay automatically select a scoring strategy. In other embodiments, thecluster engine 120 may execute several scoring strategies 442 todetermine a combined score for the cluster 252.

In an embodiment, a scoring strategy (such as scoring strategy 442)specifies an approach for scoring a cluster (such as cluster 252). Ascore may indicate a relative importance or significance of a givencluster. For example, the cluster engine 120 may execute a scoringstrategy 442-1 to determine a score by counting the number of aparticular data entity type that are included within the cluster 252.Assume, for example, a data entity corresponds with a credit account. Insuch a case, a cluster with a large number of accounts opened by asingle individual (possibly within a short time) might correlate with ahigher fraud risk. Of course, a cluster score may be related to a highrisk of fraud based on the other data in the cluster, as appropriate fora given case. More generally, each scoring strategy 442 may be tailoredbased on the data in clusters created by a given cluster strategy 230and a particular type of risk or fraud (or amounts at risk) of interestto an analyst.

According to an embodiment, the cluster engine 120 scores a cluster252-1 by first retrieving a scoring strategy 442. For example, assume ananalyst selects scoring strategy 442-1. In response, the cluster engine120 may retrieve the scoring strategy 442-1. The cluster engine 120 mayalso retrieve the cluster 252-1 from the cluster list 250. Afterdetermining the score of the cluster 252-1, the cluster engine 120 maystore the score with the cluster 252-1 in the cluster list 250.

The cluster engine 120 may score multiple clusters 252-1, 252-2 . . .252-C in the cluster list 250. The cluster engine 120 may also rank theclusters 252-1, 252-2 . . . 252-C based upon the scores. For instance,the cluster engine 120 may rank the cluster 252-1, 252-2 . . . 252-Cfrom highest score to lowest score. In various embodiment, cluster maybe ranked according into multiple scores, combinations of scores, and/ormetascores.

VII. Example User Interface

FIG. 5 illustrates an example cluster analysis user interface (UI) 500,according to one embodiment. As described above, the workflow engine 125may be configured to present the cluster analysis UI 500. As shown, theexample cluster analysis UI 500 includes a selection box 510, a clusterstrategy box 530, a cluster summary list 525, a cluster search box 520,and a cluster review window 515. The workflow engine 125 may generatethe cluster analysis UI 500 as a web application or a dynamic web pagedisplayed within the client 135.

In the example UI 500 of FIG. 5, the selection box 510 may allow theanalyst to select, for example, a seed generation strategy and/or apreviously generated seed or seed list (for example, seed list 210). Theanalyst may select the items (for example, a seed generation strategy)by, for example, entering a name of a particular item into a dropdownbox (or other interface element) in the selection box 510 (for example,the dropdown box showing a selected strategy “Strategy-A”) and selectinga “Go” button (or other interface element). Alternatively, the analystmay select a particular item by, for example, expanding the dropdown boxand selecting an item from the expanded dropdown box, which may listvarious seed generation strategies and/or seed lists, for example. Invarious examples, seed lists and/or seed generation strategies may beselected by the analyst that correspond to likely fraudulent financialaccounts, credit card account originating at a particular bank branch,savings accounts with balances above a particular amount, and/or any ofthe other seed generation strategies described below in reference to thevarious applications of the system.

For example, when the analyst selects a particular seed generationstrategy, the system may generate a seed list (for example, seed list210) and then may generate clusters based on seeds of the seed list. Theseed list and/or clusters may, in an embodiment, be generated inresponse to a selection of a particular seed generation strategy. Theseed generation strategy may generate a seed list (for example, seedlist 210) and/or clusters (for example, clusters 252-1, 252-2, . . .252-C of the cluster list 250) from the database 140 and/or an externalsource of information (for example, a cluster data source 160).Alternatively, when the analyst selects a previously generated seed orseed list (for example, seed list 210), the system may retrieve datarelated to the selected seed list (for example, the seed entities,clusters, and/or related clustered data entities) from, for example,database 140 and/or an external source of information (for example, acluster data source 160). In an embodiment, clusters may be generated inresponse to a selection of a previously generated seed list (or seed).Alternatively, cluster may be been previously generated, and may beretrieved in response to selection of a previously generated seed list(or seed). In an embodiment, the analyst may select a particular clusterof interest via the selection box 510.

Further, in the example UI 500 the cluster strategy box 530 displays thecluster strategies 232 that the cluster engine 120 ran against the seedlist 210. The cluster engine 120 may execute multiple cluster strategies232 against the seed list 210, so there may be multiple clusterstrategies 232 listed in the cluster strategy box 530. The analyst mayclick on the name of a given cluster strategy 232 in the clusterstrategy box 530 to review the clusters 252 that the cluster strategy232 generated.

In an embodiment, the workflow engine 125 generates for display in theUI 500 summaries of the clusters 252 in the cluster summary list 525.For example, the summaries may include characteristics of the clusters252, such as identifiers, scores, and/or analysts assigned to analyzethe clusters 252. The workflow engine 125 may select the clusters 252for display in the cluster summary list 525 according to those or othercharacteristics. For instance, the workflow engine 125 may display thesummaries in the order of the scores of the clusters 252, where asummary of the highest scoring cluster 252 is displayed first.

The workflow engine 125 may control the order and selection of thesummaries within the cluster summary list 525 based upon an input fromthe analyst. The cluster search box 520 may include a search text boxcoupled to a search button and a pull-down control. The analyst mayenter a characteristic of a cluster 252 in the search text box and theninstruct the workflow engine 125 to search for and display clusters 252that include the characteristic by pressing the search button. Forexample, the analyst may search for clusters with a particular score.The pull-down control may include a list of different characteristics ofthe clusters 252, such as score, size, assigned analyst, and/or datecreated. The analyst may select one of the characteristics to instructthe workflow engine 125 to present the summaries of the clusters 252arranged by that characteristic.

In an embodiment, the workflow engine 125 is also configured to presentdetails of a given cluster 252 within the cluster review window 515. Theworkflow engine 125 displays the details of the cluster 252, forexample, the score, and/or average account balances within a cluster,when the analyst clicks a mouse pointer on the associated summary withinthe cluster summary list 525. The workflow engine 125 may presentdetails of the cluster 252, such as the name of an analyst assigned toanalyze the cluster 252, a score of the cluster 252, and/or statisticsor graphs generated from the cluster 252. These details may allow theanalyst to determine whether to investigate the cluster 252 further. Thecluster review window 515 may also include a button which may be clickedto investigate a cluster 252 within a graph, and an assign button forassigning a cluster to an analyst.

An analyst may click a mouse pointer on an “Investigate in Graph” buttonrepresenting a cluster to investigate the cluster within an interactivegraph. The interactive representation may be a visual graph of thecluster 252, where icons represent the entities of the cluster 252 andlines between the icons represent the links between entities of thecluster 252. For example, the workflow engine 125 may display theinteractive graph of the cluster 252 similar to the representation ofthe cluster 252 in FIG. 3C. The interactive representation may allow theanalyst to review the attributes of the related data entities and/orperform queries for additional related data entities.

In an embodiment, an administrative user may click a mouse pointer on anassign button to assign the associated cluster 252 to an analyst. Theworkflow engine 125 may also allow the administrative user to createtasks associated with the clusters 252, while the administrative userassigns the cluster 252. For example, the administrative user may createa task for searching within the three highest scoring clusters 252 forfraudulent credit card accounts. The workflow engine 125 may display thesummaries in the cluster summary list 525 according to the names of theanalysts assigned to the clusters 252. Likewise, the workflow engine 125may only display summaries for the subset of the clusters 252 assignedto an analyst.

The interface shown in FIG. 5 is included to illustrate one exemplaryinterface useful for navigating and reviewing clusters generated usingthe cluster engine 120 and the workflow engine 125. In otherembodiments, other user interface constructs may be used to allow theanalyst to select cluster strategies 232, scoring strategies 242, and/orseed generation strategies, initiate an investigation, and/or review andanalyze the clusters 252. For example, the workflow engine 125 maydisplay additional controls within the cluster analysis UI 500 forcontrolling the cluster generation process and selecting seed generationstrategies, cluster strategies 232, and/or scoring strategies 242. Also,the UI 500 may be displayed without the selection box 510 or the optionsto select a seed generation strategy. In addition, although the workflowengine 125 generates the cluster analysis UI 500, in various embodimentsthe cluster analysis UI 500 may be generated by a software applicationdistinct from the workflow engine 125. Further, in various embodiments,the cluster review window 515 may be configured to display a preview ofthe cluster 252 and/or additional statistics generated from the cluster252. As such, an interactive representation of the cluster 252 may bepresented in an additional UI and/or the cluster 252 may be exported toanother software application for review by the analyst.

In an alternative embodiment, and as described below in reference toFIGS. 10E, 11E, 12E, and 13E, the cluster analysis user interface mayinclude a list of clusters in a first column, a list of scoresassociated with a selected cluster in a middle column, and/or detailsassociated with a selected score in a last column. Such an arrangementmay advantageously enable an analyst to investigate various scoresassociated with a cluster. Additionally, clusters in such an interfacemay advantageously be prioritized according to any of multiple scoresand/or metascores.

VIII. Example Operations

FIG. 6 is a flowchart of an example method of generating clusters,according to an embodiment. Although the method is described inconjunction with the systems of FIGS. 1 and 2, persons skilled in theart will understand that any system configured to perform the method, inany order, is within the scope of this disclosure. Further, the method600 may be performed in conjunction with method 700 for scoring acluster, described below.

As shown, example cluster generation method 600 begins at block 605,where the cluster engine 120 retrieves a cluster strategy (e.g., clusterstrategy 232-2) and a seed 212. Once a cluster strategy is selected, thecluster engine 120 may identify a list of seeds from which to buildclusters using the selected cluster strategy. At block 610, the clusterengine 120 initializes a cluster 252 with one of the seeds in the list.The cluster 252 may be stored as a graph data structure. The clusterengine 120 may initialize the graph data structure and then add the seed212-1 to the graph data structure as the first data entity.

At block 615, the cluster engine 120 may grow the cluster 252 byexecuting the search protocol of a data binding 237 from the clusterstrategy 232-2. The cluster strategy 232-2 may include a series of databindings 237 that the cluster engine 120 executes to retrieve relateddata entities. A given data binding 237 may include queries to executeagainst a cluster data source 160 using the seed as an input parameter.For example, if the seed 212-1 is an account number, then the databinding 237 may retrieve the data identifying the owner of the accountwith the account number. After retrieving this information, the clusterengine 120 may add the customer data entity to the cluster as a relateddata entity and the account owner attribute as the link between the seed212-1 and the related data entity. After retrieving the related dataentities, the cluster engine 120 may add them to the cluster 252.

At block 620, the cluster engine 120 determines if the cluster strategy232-2 is fully executed. If not the method 600 returns to block 615 toexecute additional data bindings for a given seed. Alternatively, asdescribed above, the cluster engine 120 may grow the duster by searchingfor, accessing, and/or filtering various data entities through, forexample, a generic interface to various internal and/or external datasources. Further, in an embodiment, the cluster engine 120 may determinewhether the cluster being generated is to be merged with anothercluster, as described above. Once the cluster strategy is executed forthat seed, the cluster engine 120 may determine and assign a score (ormultiple scores) to that cluster (relative to a specified scoringstrategy). After generating clusters for a group of seeds, such clustersmay be ordered or ranked based on the relative scores. Doing so mayallow an analyst to rapidly identify and evaluate clusters determined torepresent, for example, a high risk of fraud.

At block 625, the cluster engine 120 may store the cluster 252 incluster list 250. As mentioned above, the cluster list 250 may be acollection of tables within a relational database, where a table mayinclude the seed and related data entities of the cluster 252 andanother table may include links between the related data entities of thecluster 252.

At block 630, the cluster engine 120 determines if there are more seeds212 to analyze in the seed list 210. If so, the method 600 returns toblock 605 to generate another cluster from the next seed. Otherwise, themethod 600 ends. Note, while method 600 describes a single cluster beinggenerated, one of skill in the art will recognize that multipleinstances of the cluster generation process illustrated by method 600may be performed in parallel.

FIG. 7 is a flowchart of an example method of scoring clusters,according to an embodiment. Although the method is described inconjunction with the systems of FIGS. 1 and 4, persons skilled in theart will understand that any system configured to perform the methodsteps, in any order, is within the scope of the present invention.

As shown, the example cluster scoring method 700 begins at block 705,where the cluster engine 120 retrieves a scoring strategy 442 and acluster 252 (for example, a cluster just created using the method 600 ofFIG. 6). In other cases, the cluster engine 120 may retrieve the scoringstrategy 442 associated with a stored cluster. Other alternativesinclude an analyst selecting a scoring strategy 442 through the client135, the cluster engine 120 via the cluster analysis UI 500, a script,or a configuration file. The cluster engine 120 may retrieve theselected scoring strategy 442 from the scoring strategy store 440, andthe cluster 252 from the duster list 250.

At block 710, the cluster engine 120 executes the scoring strategy 442against the cluster 252. The scoring strategy 442 may specifycharacteristics of the related data entities within the cluster 252 toaggregate. The cluster engine 120 may execute the scoring strategy 442by aggregating the specified characteristics together to determine ascore. For instance, the cluster engine 120 may aggregate accountbalances of related data entities that are account data entities. Insuch a case, a total amount of dollars (or average dollars or any otheraggregated, averaged, or normal attribute of the cluster) includedwithin the balances of the account data entities of the cluster 252 maybe the score of the cluster 252.

At block 715, the cluster engine 120 may store the score with thecluster 252 in the cluster list 250. At step 720, the cluster engine 120determines if there are more clusters 252 to score. For example, in oneembodiment, a set of clusters may be re-scored using an updated scoringstrategy. In other cases, the cluster engine may score each cluster whenit is created from a seed (based on a given cluster generation andcorresponding scoring strategy). If more clusters remain to be scored(or re-scored), the method 700 returns to block 705.

At block 725, the cluster engine 120 may rank the clusters 252 accordingto the scores of the clusters 252. For example, after re-scoring a setof clusters (or after scoring a group of clusters generated from a setof seeds), the cluster engine 125 may rank the clusters 252 from highestscore to lowest score. The ranking may be used to order a display ofsummaries of the clusters 252 presented to the analyst. The analyst mayrely upon the ranking and scores to determine which clusters 252 toanalyze first. The ranking and sorting may generally be performedon-demand when an analyst is looking for a cluster to investigate. Thus,the ranking need not happen at the same time as scoring. Further, theclusters may be scored (and later ranked) using different rakingstrategies.

In various embodiments, multiple scores for each cluster may bedetermined according to methods similar to the example method 700.Accordingly, clusters may be ranked according to any of multiple scores.Additionally, in various embodiments, multiple scores may be combinedand/or aggregated into a metascore that may be used to rank theclusters. Various example score and metascore determinations aredescribed below in reference to FIGS. 10C, 11C, 12C, and 13C.

IX. Example Implementation Mechanisms/Systems

FIG. 8 illustrates components of an illustrative server computing system110, according to an embodiment. The server computing system 110 maycomprise one or more computing devices that may perform a variety oftasks to implement the various operations of the data analysis system.As shown, the server computing system 110 may include, one or morecentral processing unit (CPU) 860, a network interface 850, a memory820, and a storage 830, each connected to an interconnect (bus) 840. Theserver computing system 110 may also include an I/O device interface 870connecting I/O devices 875 (for example, keyboard, display, mouse,and/or other input/output devices) to the computing system 110. Further,in context of this disclosure, the computing elements shown in servercomputing system 110 may correspond to a physical computing system (forexample, a system in a data center, a computer server, a desktopcomputer, a laptop computer, and/or the like) and/or may be a virtualcomputing instance executing within a hosted computing environment.

The CPU 860 may retrieve and execute programming instructions stored inmemory 820, as well as store and retrieve application data residing inmemory 820. The bus 840 may be used to transmit programming instructionsand application data between the CPU 860, I/O device interface 870,storage 830, network interface 850, and memory 820. Note that the CPU860 is included to be representative of, for example, a single CPU,multiple CPUs, a single CPU having multiple processing cores, a CPU withan associate memory management unit, and the like.

The memory 820 is included to be representative of, for example, arandom access memory (RAM), cache and/or other dynamic storage devicesfor storing information and instructions to be executed by CPU 860.Memory 820 also may be used for storing temporary variables or otherintermediate information during execution of instructions to be executedby CPU 860. Such instructions, when stored in storage media accessibleto CPU 860, render server computing system 110 into a special-purposemachine that is customized to perform the operations specified in theinstructions.

The storage 830 may be a disk drive storage device, a read only memory(ROM), or other static, non-transitory, and/or computer-readable storagedevice or medium coupled to bus 840 for storing static information andinstructions for CPU 860. Although shown as a single unit, the storage830 may be a combination of fixed and/or removable storage devices, suchas fixed disc drives, removable memory cards, and/or optical storage,network attached storage (NAS), and/or a storage area-network (SAN).

Programming instructions, such as the cluster engine 120 and/or theworkflow engine 125, may be stored in the memory 820 and/or storage 830in various software modules, The modules may be stored in a mass storagedevice (such as storage 830) as executable software codes that areexecuted by the server computing system 110. These and other modules mayinclude, by way of example, components, such as software components,object-oriented software components, class components and taskcomponents, processes, functions, attributes, procedures, subroutines,segments of program code, drivers, firmware, microcode, circuitry, data,databases, data structures, tables, arrays, and variables.

Illustratively, according to an embodiment, the memory 820 stores a seedlist 210, a cluster engine 120, a cluster list 250, and a workflowengine 125 (as described with reference to the various figures above).The cluster engine 120 may include a cluster strategy 232-2. Theparticular cluster strategy 232-2 may include data bindings 237-1,237-2, and 237-3, with which the cluster engine 120 may access thecluster data source 160. The workflow engine 125 may include a scoringstrategy 442-1.

Illustratively, according to an embodiment, the storage 830 includes acluster strategy store 230, data bindings store 835, and a scoringstrategy store 440. As described above, the cluster strategy store 230may include a collection of different cluster strategies 232, such ascluster strategy 232-2. For example, the cluster strategy store 230 maybe a directory that includes the cluster strategies 232-1, 232-2 . . .232-N as distinct modules. The scoring strategy store 440 may include acollection of different scoring strategies 442, such as scoring strategy442-2, and may also be a directory of distinct modules. The data bindingstore 835 may include data bindings 237-1, 237-2 . . . 237-M, which mayalso be stored as distinct modules within a directory.

Although shown in memory 820, the seed list 210, cluster engine 120,cluster list 250, and workflow engine 125, may be stored in memory 820,storage 830, and/or split between memory 820 and storage 830. Likewise,copies of the cluster strategy 232-2, data binding 237-1, 237-2, and237-3, and scoring strategy 442-2 may be stored in memory 820, storage830, and/or split between memory 820 and storage 830.

The network 150 may be any wired network, wireless network, orcombination thereof. In addition, the network 150 may be a personal areanetwork, local area network, wide area network, cable network, satellitenetwork, cellular telephone network, or combination thereof. Protocolsand components for communicating via the Internet or any of the otheraforementioned types of communication networks are well known to thoseskilled in the art of computer communications and thus, need not bedescribed in more detail herein.

As described above, in various embodiments the system may be accessibleby an analyst (or other operator or user) through a web-based viewer,such as a web browser. In this embodiment, the user interface may begenerated by the server computing system 110 and transmitted to the webbrowser of the analyst. Alternatively, data necessary for generating theuser interface may be provided by the server computing system 110 to thebrowser, where the user interface may be generated. The analyst/user maythen interact with the user interface through the web-browser. In anembodiment, the user interface of the data analysis system may beaccessible through a dedicated software application. In an embodiment,the client computing device 130 may be a mobile computing device, andthe user interface of the data analysis system may be accessible throughsuch a mobile computing device (for example, a smartphone and/ortablet). In this embodiment, the server computing system 110 maygenerate and transmit a user interface to the mobile computing device.Alternatively, the mobile computing device may include modules forgenerating the user interface, and the server computing system 110 mayprovide user interaction data to the mobile computing device. In anembodiment, the server computing system 110 comprises a mobile computingdevice. Additionally, in various embodiments any of the componentsand/or functionality described above with reference to the servercomputing system 110 (including, for example, memory, storage, CPU,network interface, I/O device interface, and the like), and/or similaror corresponding components and/or functionality, may be included in theclient computing device 130.

According to various embodiments, the data analysis system and othermethods and techniques described herein are implemented by one or morespecial-purpose computing devices. The special-purpose computing devicesmay be hard-wired to perform the techniques, or may include digitalelectronic devices such as one or more application-specific integratedcircuits (ASICs) or field programmable gate arrays (FPGAs) that arepersistently programmed to perform the techniques, or may include one ormore general purpose hardware processors programmed to perform thetechniques pursuant to program instructions in firmware, memory, otherstorage, or a combination. Such special-purpose computing devices mayalso combine custom hard-wired logic, ASICs, or FPGAs with customprogramming to accomplish the techniques. The special-purpose computingdevices may be desktop computer systems, server computer systems,portable computer systems, handheld devices, networking devices or anyother device or combination of devices that incorporate hard-wiredand/or program logic to implement the techniques.

Computing devices of the data analysis system may generally becontrolled and/or coordinated by operating system software, such as iOS,Android, Chrome OS, Windows XP, Windows Vista, Windows 7, Windows 8,Windows Server, Windows CE, Unix, Linux, SunOS, Solaris, iOS, BlackberryOS, VxWorks, or other compatible operating systems. In otherembodiments, the computing devices may be controlled by a proprietaryoperating system. Conventional operating systems control and schedulecomputer processes for execution, perform memory management, providefile system, networking, I/O services, and provide a user interfacefunctionality, such as a graphical user interface (“GUI”), among otherthings.

In general, the word “module,” as used herein, refers to a collection ofsoftware instructions, possibly having entry and exit points, written ina programming language, such as, for example, Java, Lua, C or C++. Asoftware module may be compiled and linked into an executable program,installed in a dynamic link library, or may be written in an interpretedprogramming language such as, for example, BASIC, Perl, or Python. Itwill be appreciated that software modules may be callable from othermodules or from themselves, and/or may be invoked in response todetected events or interrupts. Software modules configured for executionon computing devices may be provided on a computer readable medium, suchas a compact disc, digital video disc, flash drive, magnetic disc, orany other tangible medium, or as a digital download (and may beoriginally stored in a compressed or installable format that requiresinstallation, decompression or decryption prior to execution). Suchsoftware code may be stored, partially or fully, on a memory device ofthe executing computing device, for execution by the computing device.Software instructions may be embedded in firmware, such as an EPROM. Itwill be further appreciated that hardware devices (such as processorsand CPUs) may be comprised of connected logic units, such as gates andflip-flops, and/or may be comprised of programmable units, such asprogrammable gate arrays or processors. The modules or computing devicefunctionality described herein are preferably implemented as softwaremodules, but may be represented in hardware devices. Generally, themodules described herein refer to software modules that may be combinedwith other modules or divided into sub-modules despite their physicalorganization or storage.

Server computing system 110 may implement various of the techniques andmethods described herein using customized hard-wired logic, one or moreASICs or FPGAs, firmware and/or program logic which, in combination withvarious software modules, causes the server computing system 110 to be aspecial-purpose machine. According to one embodiment, the techniquesherein are performed by server computing system 110 in response to CPU860 executing one or more sequences of one or more modules and/orinstructions contained in memory 820. Such instructions may be read intomemory 820 from another storage medium, such as storage 830. Executionof the sequences of instructions contained in memory 820 may cause CPU840 to perform the processes and methods described herein. Inalternative embodiments, hard-wired circuitry may be used in place of orin combination with software instructions.

The term “non-transitory media,” and similar terms, as used hereinrefers to any media that store data and/or instructions that cause amachine to operate in a specific fashion. Such non-transitory media maycomprise non-volatile media and/or volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks, such as storage 830.Volatile media includes dynamic memory, such as memory 820. Common formsof non-transitory media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge, and networkedversions of the same.

Non-transitory media is distinct from but may be used in conjunctionwith transmission media. Transmission media participates in transferringinformation between non-transitory media. For example, transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise bus 840. Transmission media may also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to CPU 860 for execution. For example, theinstructions may initially be carried on a magnetic disk or solid statedrive of a remote computer. The remote computer may load theinstructions and/or modules into its dynamic memory and send theinstructions over a telephone or cable line using a modem. A modem localto server computing system 820 may receive the data on thetelephone/cable line and use a converter device including theappropriate circuitry to place the data on bus 840. Bus 840 carries thedata to memory 820, from which CPU 860 retrieves and executes theinstructions. The instructions received by memory 820 may optionally bestored on storage 830 either before or after execution by CPU 860.

X. Additional Example Applications

While financial fraud using credit card accounts is used as a primaryreference example in the discussion above, the techniques describedherein may be adapted for use with a variety of data sets and in variousapplications. For example, information from data logs of online systemsmay be evaluated as seeds to improve cyber security. In such a case, aseed may be a suspicious IP address, a compromised user account, and thelike. From the seeds, log data, DHCP logs, IP blacklists, packetcaptures, webapp logs, and other server and database logs may be used tocreate clusters of activity related to the suspicions seeds. Otherexamples include data quality analysis used to cluster transactionsprocessed through a computer system (whether financial or otherwise). Anumber of examples of such applications are described in detail below inreference to, for example, FIG. 9, FIGS. 10A-10E (tax fraud detection),FIGS. 11A-11E (beaconing malware detection), 12A-12E (malware user-agentdetection), and FIGS. 13A-13E (activity trend detection).

XI. Example Generalized Application of the Data Analysis System

FIG. 9 is a flowchart of an example generalized method of the dataanalysis system, according to various embodiments of the presentdisclosure. In various embodiments, fewer blocks or additional blocksmay be included in the process, or various blocks may be performed in anorder different from that shown in the figure. In an embodiment, one ormore blocks in the figure may be performed by various components of thedata analysis system, for example, server computing system 110(described above in reference to FIG. 8).

As described above, and as shown in FIG. 9, the data analysis system maygenerate a seed or multiple seeds (block 910), may generate clustersbased on those seed(s) (block 920), may generate a score or multiplescores for each generated cluster (block 930), may generate a metascorefor each generated cluster (block 940), and may optionally rank thegenerated clusters based on the generated metascores (block 950). Invarious embodiments, the data analysis system may or may not generatemultiple scores for each cluster, may or may not generate metascores foreach cluster, and/or may or may not rank the clusters. In an embodiment,the system may rank clusters based on one or more scores that are notmetascores.

Further, as described above, the seeds may include one or multiple dataentities, and may be generated based on seed generation strategiesand/or rules. Similarly, the clusters may include one or multiple dataentities related to a seed, including the seed, and may be generatedbased on cluster generation strategies and/or rules (including databindings and/or searching and filtering are performed through, forexample, a generic interface to various data sources). Scores andmetascores may be determined based on attributes, characteristics,and/or properties associated with data entities that make up a givencluster.

Four example applications of the data analysis system are describedbelow (FIGS. 10A-10E (tax fraud detection), FIGS. 11A-11E (beaconingmalware detection), 12A-12E (malware user-agent detection), and FIGS.13A-13E (activity trend detection)) in which various example methods ofseed generation, cluster generation, cluster scoring and metascoring,and cluster ranking are described. The four example applications are notintended to be limiting of the scope of the described methods, systems,and processes of the present disclosure. With respect to each of theflowcharts described below (for example, the flowcharts of FIGS.10A-10C, 11A-11C, 12A-12C, and 13A-13C), it is to be understood that, invarious embodiments, fewer blocks or additional blocks may be includedin the example processes depicted, or various blocks may be performed inan order different from that shown in the figures. Further, in variousembodiments, one or more blocks in the figures may be performed byvarious components of the data analysis system, for example, servercomputing system 110 (described above in reference to FIG. 8) and/oranother suitable computing system.

XII. Example Application of the Data Analysis System to Tax FraudDetection

FIGS. 10A-10E depict various aspects of the data analysis system asapplied to detecting tax fraud. For example, the data analysis systemmay be used in conjunction with tax-related data entities to detectfraudulent tax-related activity. Fraudulent tax-related activity mayinclude, for example, a fraudster stealing another's identity and filingon their behalf to fraudulently extract tax return money from agovernment. The tax return money may be automatically deposited in abank account and then retrieved by the fraudster. The data analysissystem may be used to detect and prevent such fraudulent activity.

According to various embodiments, and as described below, tax-relateddata entity seeds may be generated by the system as described below inreference to FIG. 10A. Each of these tax-related entity seeds mayinclude one or more tax-related entities, and the seeds may be generatedbased on a likelihood that the seeds represent fraudulent activitiesand/or data. Each of the seeds may be used as a basis for clusteringvarious other tax-related entities, as described in reference to FIGS.10B and 10D. Accordingly, the generated clusters may represent variousdata entities that are all related to potentially fraudulent activity.Each of the generated clusters may then be scored according to variouscriteria (or rules), as described below in reference to FIG. 10C. Thevarious scores and metascores generated by the system may provideindications to an analyst regarding the likelihood that the clusterincludes entities representing fraudulent activities and/or data. Theinformation, including the clusters and scores generated by the dataanalysis system, may be presented to an analyst via a user interface asdescribed below in reference to FIG. 10E. Advantageously, according toan embodiment, the analyst may sort the clusters according to theirdetermined scores (and/or metascores) so as to prioritize investigationsinto potential fraud. Further, the data analysis system mayadvantageously automatically cluster or group many related data entitiesto enable rapid investigation and evaluation by an analyst to detectlikely tax fraud.

a. Tax Fraud Detection: Seed Generation

FIG. 10A is a flowchart of an example of a seed generation method 910 aof the data analysis system as applied to tax fraud detection, accordingto various embodiments of the present disclosure. The seed generationmethod 910 a may generally be understood to correspond to block 910(seed generation) of the generalized process of FIG. 9.

Referring to FIG. 10A, at block 1012, tax-related data and/or dataentities may be received and/or accessed by the data analysis system.For example, tax-related data entities may include tax returns (1012 a),pay statements (1012 b), and/or other third-party data (1012 c), just toname a few. Such tax-related data may be accessed and/or received from,for example, various cluster data sources 160 (as shown in FIG. 1).

At block 1014, various tax fraud indicators may be received and/oraccessed by the data analysis system. Tax fraud indicators may include,for example, various data, rules, and/or criteria that may be comparedwith the tax-related data entities to determine that a tax-relatedentity is likely associated with fraudulent activity. Examples of taxfraud indicators may include Internet Protocol (IP) addresses that arefrequently used, for example, to file multiple tax returns in a shortperiod of time (1014 a); financial institution accounts that receivemultiple tax refunds (1014 b); and/or locations (such as physicaladdresses) associated with and/or receiving multiple tax refunds (1014c); just to name a few. Other tax fraud indicators may include, forexample, various of the cluster scoring criteria described below inreference to FIG. 10C. Such tax fraud indicators may be derived fromautomatic or manual analysis of tax-related data. For example, the dataanalysis system may automatically process large amounts of tax-returnsacross multiple years to determine frequently used IP addresses, bankaccounts that receive multiple tax refunds, and/or physical addressesthat are frequently used to receive tax refunds. Tax fraud indicatorsmay also be derived from known lists, and/or may be provided by ananalyst or other user of the data analysis system. In variousembodiments, tax fraud indicators may be referred to as business rules.Further, in various embodiments, the tax fraud indicators definesuspicious characteristics that may be associated with tax-related dataentities that may indicate possible fraudulent activity.

At block 1016, the tax-related data may be analyzed in view of, orcompared to, the tax fraud indicators, and particular tax-related dataentities that may be associated with tax fraud are determined. Forexample, a tax return data entity may include a return address that is aknown tax fraud indicator. In this example, the tax return data entitymay be determined by the system possibly to be related to tax fraud. Inanother example, a tax return may be filed from a particular IP addressknown to be a tax fraud indicator, and/or may include a bank accountnumber that is a known tax fraud indicator. In this example, the taxreturn may again be determined by the system to be possibly related totax fraud.

At block 1018, the tax-related entities that are determined to bepossibly related to tax fraud may be used as seeds by the data analysissystem. Accordingly, the method 910 a may identify data entity seedsthat may be used by the data analysis system in a tax-fraud detectionapplication.

b. Tax Fraud Detection: Cluster Generation

Turning now to FIG. 10B, a flowchart of an example of a clusteringmethod 920 a of the data analysis system as applied to tax frauddetection is shown, according to various embodiments of the presentdisclosure. The clustering method 920 a may generally be understood tocorrespond to block 920 (cluster generation) of the generalized processof FIG. 9. Additionally, the clustering method 920 a may correspond to aclustering strategy, as described above. In the flowchart of FIG. 10B,block 1022 indicates that each of the following blocks (1024, 1026) maybe performed for each of the seeds generated by the seed generationmethod 910 a of FIG. 10A.

At block 1024, any tax-return data entities that are related to the seedmay be clustered. Clustering of data entities may be accomplished asgenerally described above, in which data bindings are executed and/orsearching and filtering are performed (through, for example, a genericinterface to various data sources) as part of a clustering strategy.Additionally, as described above, clustered data entities may be relatedby, for example, sharing the same or similar properties,characteristics, and/or metadata. For example, assume a seed is a bankaccount. In block 1024, any tax returns that include mention of the bankaccount may be clustered with the bank account. This example may be seenvisually in FIG. 10D, which illustrates an example growth of a clusterof related data entities in a tax fraud detection application, accordingto an embodiment of the present disclosure. In FIG. 10D, boxes indicatedata entities, while lines between boxes indicate links that connectdata entities. In the example of FIG. 10D, a seed bank account 1050 hasbeen generated (such as by the process of FIG. 10A). Then, in aclustering step corresponding to block 1024 (of FIG. 10B) andrepresented by the internal cluster dashed line, a related tax return1051 that references the seed bank account is added to the cluster.

Returning to FIG. 10B, at block 1026, any other data entities related tothe clustered tax return(s) or the seed may be added to the cluster.Other related data entities may include for example, persons (1026 a),physical addresses (1026 b), and/or IP addresses (1026 c), just to namea few. An example is illustrated visually in FIG. 10D where multipledata entities have been added to the cluster, as indicated by theexternal cluster dashed line. For example, the data analysis system hasclustered two persons 1054 and 1056 (who may be, for example, a husbandand wife referenced by the tax return), a W-2 1052 (which may be adocument including employment related data), an IP address 1058 (fromwhich the tax return 1051 may have been electronically filed), anotherbank account 1060, and a physical address 1062. Additionally, the dataanalysis system has determined that both the person 1054 and the W-21052 are related (for example, the name of person 1054 may be referencedby the W-2 1052), the two persons 1054 and 1056 are related (forexample, the tax return 1051 may indicate that they are husband andwife), and the person 1056 is related to the bank account 1050.

Returning again to FIG. 10B, dashed line 1028 indicates that the clustergeneration method may optionally repeat multiple times until, forexample, the clustering strategy is completed and/or no additionalrelated data entities are found by the system. For example, in referenceto FIG. 10D, additional data entities may be clustered including anothertax return 1064 related to bank account 1060 and/or a business 1066related to the W-2 1052 and/or the person 1054. As indicated by ellipses1071, 1072, and 1073, additional data entities may be clustered insubsequent clustering steps. Further, referring to FIG. 10B, and asdescribed above, at 1028 various clusters of data entities mayoptionally be merged and/or collapsed when common data entities and/orproperties are determined between the various clusters. For example, thesystem may determine that two different generated clusters both includeperson 1056. Accordingly, the system may merge the two clusters eachincluding the common data entity (person 1056) into a single cluster.

Returning to FIG. 10B, in an embodiment the clustering method 920 a mayproceed without block 1024 (cluster related tax returns), such that anydata entities (including tax returns) related to the seed areiteratively clustered. Accordingly, in an embodiment the clusteringmethod 920 a may iteratively cluster related data entities.

Additionally, in an embodiment a cluster graph similar to the clusterillustration of FIG. 10D may be made available to an analyst or otheruser of the data analysis system. For example, an analyst may select abutton (for example, an “Investigate in Graph” button) in a userinterface of the system to view a cluster graph of a selected cluster.

c. Tax Fraud Detection: Cluster Scoring

Turning now to FIG. 10C, a flowchart of example cluster scoring methods930 a, 940 a of the data analysis system as applied to tax frauddetection is shown, according to various embodiments of the presentdisclosure. The clustering scoring methods 930 a, 940 a may generally beunderstood to correspond to blocks 930 and 940 (cluster score andmetascore generation) of the generalized process of FIG. 9.Additionally, the clustering scoring methods 930 a, 940 a may correspondwith scoring strategies, as described above. In the flowchart of FIG.10C, block 1032 indicates that each of the following blocks (1034, 1036,and 1042) may be performed for each of the clusters generated by thecluster generation method 920 a of FIG. 10B.

At block 1034, the data analysis system may access and/or receive taxfraud scoring criteria. The tax fraud scoring criteria may include anynumber of rules or scoring strategies such that multiple scores may begenerated for each cluster. Several non-limiting examples of tax fraudscoring criteria may include: a number of tax returns in the clusterknown to be fraudulent; a presence of particular types of physicaladdresses (for example, post office boxes and/or private deliveryservice mail boxes); a total number of tax returns in the cluster; asimilarity between requested tax return amounts among the tax returns inthe cluster; a number of tax returns in the cluster associated withpersons having a recent change in address and/or bank account; aninclusion in the cluster of persons under a particular age and/orpersons known to be dead; tax returns including requests for particulartypes of credits in the cluster; a number of persons in the cluster thathave never before filed a tax return; and/or a number of known fraudaccounts in the cluster.

At block 1036, the tax fraud scoring criteria may be applied to theclusters and cluster scores may be generated. In an embodiment, eachcluster score may include an absolute value and/or a weighted value. Forexample, a given cluster may include two known fraudulent tax returns.When a scoring criteria is applied that indicates a known number offraudulent tax returns, the absolute value of the score generated wouldbe two (2), as the cluster includes two such tax returns. However, thescoring criteria may also include a relative weighting that indicatesthe importance of the particular score (“a number of known fraudulenttax returns”) to an overall evaluation of the cluster in the context oftax fraud. Such a relative weighting, in one embodiment, may be a numberbetween 0 and 1, where a number closer to one indicates a greaterimportance. For example, if the “number of known fraudulent tax returns”score is considered relatively important to an evaluation of thecluster, it may be given a relative weight of, for example, “0.7”. Onthe other hand, a relatively less important score/consideration (forexample, a number of first time filers) may be given a relative weightof, for example, “0.15”. Then, when a score is calculated by the dataanalysis system, the relative weight may be multiplied by the absolutevalue to arrive at a corresponding relative value. For example, in the“number of fraudulent returns” example, where an absolute value of 2 wasdetermined, a relative value may be determined by multiplying 2 by 0.7,to arrive at a relative value of 1.4. In other embodiments, variousother methods may be employed by the data analysis system to determinerelative values of cluster scores. For example, as may be recognized byone of skill in the art, the system may normalize the absolute values ofeach of the scores before applying a relative weighting to arrive at aweighted value. Examples of cluster scores presented to an analyst orother user of the data analysis system are shown and described below inreference to FIG. 10E. In an embodiment, the importance of particularscores may be determined and/or based on empirical determinations and/orpast cluster evaluations. For example, over time the system and/or ananalyst may determine, based on evaluations of clusters by analysts,that particular scores are better indicators of fraud than others. Suchbetter indicators may be determined to be more important, and maytherefore be weighted more heavily by the system.

At block 1042, a metascore may be generated for the clusters. Thecluster metascore may be based on a combination or aggregation of theindividual scores generated in block 1036. Alternatively, the metascoresmay be separately determined scores. In an embodiment, a metascore maybe calculated by summing, multiplying, and/or otherwise aggregating oraveraging the various individual scores together. The metascore may, inan embodiment, capture the relative importance of each of the individualscores by weighting each of the individual scores in a manner similar tothat described above with reference to block 1036.

In various embodiments, metascores and/or scores may advantageouslyenable an analyst to directly compare and/or prioritize variousclusters, and/or may advantageously be used by the data analysis systemto prioritize a list of clusters and/or scores related to a cluster.

d. Tax Fraud Detection: Example User Interface

FIG. 10E illustrates an example cluster analysis user interface of thedata analysis system as applied to tax fraud detection, according to anembodiment of the present disclosure. The example user interface of FIG.10E includes a list of clusters 1082, a list of scores 1084, and adetailed view of a score 1086. In various embodiments, more or fewerelements may be included in the user interface, and/or the elements maybe arranged differently. Additionally, while certain aspects of the userinterface of FIG. 10E may be similar to those of the user interface ofFIG. 5 described above, the user interface of the FIG. 10E includes anumber of differences. For example, differing from the user interface ofFIG. 5, the user interface of the FIG. 10E may include a list ofclusters in a first column, a list of scores associated with a selectedcluster in a middle column, and/or details associated with a selectedscore in a last column. Such an arrangement may advantageously enable ananalyst to investigate various scores associated with a cluster.Additionally, clusters in such an interface may advantageously beprioritized according to any of multiple scores and/or metascores, asdescribed above.

In the example user interface of FIG. 10E, an analyst or user hasselected “Tax Fraud Cluster 1.” Accordingly, various scores associatedwith that cluster may be displayed in the list of scores 1084. Forexample, scores are listed for “Known fraudulent returns” and “Returnsw/similar values,” among others. Additionally, in the example userinterface, the analyst has selected the “Returns w/similar values”score. Accordingly, details related to that score may be displayed inthe detailed view 1086.

According to an embodiment, various items of information may be includedin the user interface that may be useful to an analyst in evaluatingand/or investigating the generated clusters. For example, metascoresassociated with each of the generated clusters may be shown in the listof clusters 1082, and/or the clusters may be prioritized according tothe metascores. In another example, absolute values (1090) and/orweighted values (1088) may be displayed in the list of scores 1084 foreach score. In the example shown, a metascore of “0.5” may be calculatedfor “Tax Fraud Cluster 1” by, for example, averaging the various clusterscores (for example, (1.0+0.8+0.6+0.6+0+0)/6=0.5). In another example,the detailed view 1086 may include a graph that shows additionalinformation related to the selected score. For example, in FIG. 10E, thegraph shown in the detailed view 1086 shows a distribution of the valuesof tax return requests associated with each of the tax returns in thecluster. In the example, both of the tax returns in the cluster includeda request for a tax return of $2,000. In other embodiments, variousother detailed information may be included in the user interface of FIG.10E.

In an embodiment, the data analysis system may automatically evaluatethe generated clusters to determine a likelihood of tax fraud. Forexample, the system may determine that a cluster having a metascorebelow a particular threshold is likely not fraud, while a cluster havinga metascore above another particular threshold likely is fraud. In anembodiment, the system may determine that a cluster having a metascorewithin a particular range of thresholds requires additional analysis byan analyst as the likelihood of fraud is not conclusive. In anembodiment, an analyst may adjust the thresholds, the metadatacalculations, and/or the weighting applied to the scores. Further, theanalyst may mark various clusters as, for example, fraudulent, likelyfraudulent, likely not fraudulent, and/or not fraudulent. Additionally,the analyst may dispatch other analysts to review particular clustersand/or mark particular clusters for further analysis.

XIII. Example Application of the Data Analysis System to BeaconingMalware Detection

FIGS. 11A-11E depict various aspects of the data analysis system asapplied to detecting beaconing malware. For example, the data analysissystem may be used in conjunction with beaconing malware-related dataentities to detect malware on a computer system or a network of computersystems. Beaconing malware activity may include, for example, a softwareprogram maliciously installed on a target computer system thatperiodically attempts to transmit data and/or communicate with a remotecomputer system. Typically, beaconing malware may attempt connections ona regular, well-defined and periodic basis, where the time betweenattempts is on the order of hours, days, weeks, or months. Such amalicious software program may be dormant (with the exception ofbeaconing activity) for a period of time before it is activated by theremote computer system. Once activated, the malicious software programmay perform various malicious actions including, for example, accessing,modifying, and/or deleting files; extracting personal data andinformation; obtaining passwords and usernames; and the like.Accordingly, it may be important to detect such beaconing malware sothat it may be removed before it is activated. The data analysis systemmay be used to detect such beaconing malware, as described below.

According to various embodiments, beaconing malware-related data entityseeds (referred to herein as “beaconing seeds”) may be generated by thesystem as described below in reference to FIG. 11A. Each of thesebeaconing seeds may include pairs of beaconing entities (referred to a“beaconing pairs”), such as a beacon originator and a beacon recipient,and the seeds may be generated based on a likelihood that the seedsrepresent beaconing activities and/or data. Each of the seeds may beused as a basis for clustering various other beaconing malware-relatedentities, as described in reference to FIGS. 11B and 11D. Accordingly,the generated clusters may represent various data entities that are allrelated to potential beaconing malware-related activity. Each of thegenerated clusters may then be scored according to various criteria (orrules), as described below in reference to FIG. 11C. The various scoresand metascores generated by the system may provide indications to ananalyst regarding the likelihood that the cluster includes entitiesrepresenting beaconing activities and/or data. The information,including the clusters and scores generated by the data analysis system,may be presented to an analyst via a user interface as described belowin reference to FIG. 11E. Advantageously, according to an embodiment,the analyst may sort the clusters according to their determined scores(and/or metascores) so as to prioritize investigations into potentialbeaconing malware. Further, the data analysis system may advantageouslyautomatically cluster or group many related data entities to enablerapid investigation and evaluation by an analyst to detect likelybeaconing malware.

In an embodiment, and as described below, the data analysis system maybe used in a network environment in which an internal network is incommunication with an external network. The system may be used todetermine whether any computer systems of the internal network have beeninfected by beaconing malware that is communicating with computersystems of the external network. Various computerized devices may beincluded in the internal network that may be capable to capturing and/orlogging data traffic between the internal network and the externalnetwork including, for example, network routers and/or switches.

a. Beaconing Malware Detection: Seed Generation

FIG. 11A is a flowchart of an example of a seed generation method 910 bof the data analysis system as applied to beaconing malware detection,according to various embodiments of the present disclosure. The seedgeneration method 910 b may generally be understood to correspond toblock 910 (seed generation) of the generalized process of FIG. 9.

Referring to FIG. 11A, at block 1112, network communications and/or datatraffic information between the internal and external networks may becaptured by the data analysis system. Various items of information maybe captured including, for example, external IP addresses contacted(1112 a), external domains contacted (1112 b), internal IP addressescontacting the external IP addresses and domains (1112 c), and the like.These items of information may be captured by, for example, a networktraffic router that connects the internal and external networks to oneanother. The network traffic router may, for example, log such items ofinformation such that they may be read and analyzed by the data analysissystem. Alternatively, the network traffic may be captured by, forexample, other types of computerized sensors. Each of the abovedescribed items of information may be a data entity in the context ofthe data analysis system.

At block 1113, the system may generate internal-external connectionpairs. Each of the internal-external connection pairs may include aparticular internal IP address and a particular external IP addressand/or domain that was contacted by the internal IP address. At block1114, time series of the generated internal-external connection pairsmay be generated. For example, the system may determine sets ofconnection pairs that have common internal IP addresses and external IPaddresses or domains. Then, for each set, a time series may be generatedthat represents each point in time that the same or a similar connectionis made between a particular internal IP address and external IP addressor domains. Each of the time series may span a particular time period.For example, each time series may span a number of days, weeks, months,or years. Thus, a connection pair time-series (or simply “connectionpair series” or “connection series”), may indicate multiple connectionsmade between a particular internal and external IP address (or domain orother device identifier) and/or a periodicity or other patternindicating when the connections were made. The internal-externalconnection pairs may be plotted along each time series for theparticular time period.

At block 1116, the data analysis system may filter out any noise in eachtime series. For example, the connection pairs in each connection seriesmay be analyzed in order to identify any connection pairs of theparticular connection series that should be indicated as noise. Noise ina connection series may include, for example, any internal-externalconnection pairs that have a low likelihood of being related tobeaconing activity and/or to malicious activity. Various filter criteriamay be applied to filter out noise. Examples of noise filtering criteriamay include, but are not limited to: filter 1116 a, which detectsfrequently established connections, such as the same or similarconnection pairs (for example, multiple connection pairs from the sameinternal IP to the same external IP and/or domain) that occur with shortintervals (or deltas) of time between them (for example, intervals onthe order of seconds, or intervals that are shorter than are typicallyemployed by beaconing malware); filter 1116 b, which detects connectionpairs that have only been occurring for a short period of time (forexample, for a week or less); filter 1116 c, which detects connectionpairs with popular or well-known legitimate external domains (forexample, a third-party produced list of popular domains may be used bythe system); and/or filter 1116 d, which detects connection pairs madeby legitimate software for, for example, software updates (in anembodiment, this filter criteria may be applied on a per-computer systembasis, such that a determination may be made regarding the legitimacy ofparticular pieces of software on each individual computer system).

Once connection pairs that include noise, or which are not likelyrelated to beaconing malware, are filtered from each connection series,at block 1117 a beaconing score may be computed for each connection pairseries. A beaconing score may be computed in any of various ways. Oneexample of computing a beaconing score is shown in block 1117 a. In theexample of block 1117 a, the system may calculate a variance of theparticular connection pair series. The variance may, for example,provide an indication of the regularity, or periodicity, of theconnection pairs over time. Higher variances may indicate that theconnection pair is less likely to be related to malware beaconingactivity, as malware beaconing activity may generally occur at veryregular intervals. Thus, lower variances may indicate that theconnection pair is more likely to be related to malware beaconingactivity. Another example of computing a beaconing score is shown inblock 1117 b. In the example of block 1117 b, the system may calculate amean of the particular connection pair series. The mean may, forexample, provide an indication of the average time between eachconnection pair over time. Particular mean values, for example, aparticular number of days, weeks, and/or months, may indicate higher orlower likelihood that the connection series is related to malwarebeaconing activity. In another example, some combination of a varianceand a mean of a connection pair series may be used by the system as abeaconing score (for example, a variance divided or normalized by a meanor a mean squared). In an embodiment, the variance is calculated basedon an average of squared differences from the mean time betweenconnections in a time series.

At block 1118, the system may determine which connection pairs havebeaconing scores that satisfy a particular threshold. For example, thesystem may determine that any beaconing pairs having beaconing scoresbelow a particular variance are likely to represent malware beaconingactivity. Accordingly, the data analysis system may designate and usethose connection pairs as seeds. Thus, the method 910 b may be used togenerate seeds including a connection pair (e.g., an internal IP addressand an external IP address or domain) that may be used by the dataanalysis system in a beaconing malware detection application.

b. Beaconing Malware Detection: Cluster Generation

Turning now to FIG. 11B, a flowchart of an example of a clusteringmethod 920 b of the data analysis system as applied to beaconing malwaredetection is shown, according to various embodiments of the presentdisclosure. The clustering method 920 b may generally be understood tocorrespond to block 920 (cluster generation) of the generalized processof FIG. 9. Additionally, the clustering method 920 b may correspond to aclustering strategy, as described above. In the flowchart of FIG. 11B,block 1122 indicates that the following block (1124) may be performedfor each of the seeds generated by the seed generation method 910 b ofFIG. 11A.

At block 1124, any data entities that are related to the seed may beclustered. Clustering of data entities may be accomplished as generallydescribed above, in which data bindings are executed and/or searchingand filtering are performed (through, for example, a generic interfaceto various data sources) as part of a clustering strategy. Additionally,as described above, clustered data entities may be related by, forexample, sharing the same or similar properties, characteristics, and/ormetadata. Examples of data entities that may be clustered include, butare not limited to: users (for example, persons having accounts onparticular computer systems), internal IP addresses, internal IPaddresses that connect to external domains, internal computer systems,internal computer systems that connect to external domains, external IPaddresses, external domains, external IP addresses associated withexternal domains, other data feed data entities (for example, dataentities drawn from public and/or private whitelists or blacklists, suchas data entities representing known bad domains, known good domains,known bad IP addresses, and the like), host-based events (such as, forexample, virus scan alerts and/or logged events, intrusion preventionsystem alerts and/or logged events, and the like), and the like.

FIG. 11D illustrates an example growth of a cluster of related dataentities in a beaconing malware detection application, according to anembodiment of the present disclosure. In FIG. 11D, boxes indicate dataentities, while lines between boxes indicate links that connect dataentities. As described above, seeds in the described beaconing-malwaredetection application of the data analysis system may be connectionpairs. As shown in the example of FIG. 11D, a seed connection pair hasbeen generated (such as by the process of FIG. 11A) that includes aninternal IP address 1152 and an external domain 1154, as indicatedvisually by the internal seed dashed line 1150. Then, in a clusteringstep corresponding to block 1124 (of FIG. 11B) and represented by theexternal cluster dashed line 1166, various other data entities relatedto the seed data entities may be added to the cluster. For example, thedata analysis system has clustered an internal computer system 1156(that may be associated with the internal IP address 1152), a user 1158(who may be a user of the computer system 1156 at the internal IPaddress 1152), and two other computer systems 1160 and 1162 that haveeach also connected to the external domain 1154.

Returning again to FIG. 11B, dashed line 1126 indicates that the clustergeneration method may optionally repeat multiple times until, forexample, the clustering strategy is completed and/or no additionalrelated data entities are found by the system. For example, in referenceto FIG. 11D, additional data entities may be clustered includinghost-based events 1167 and 1168 associated with the computer system1156, and users 1172 and 1174 of the computer system 1160. As indicatedby ellipses 1164, 1170 and 1176, additional data entities may beclustered in subsequent clustering steps. Further, referring to FIG.11B, and as described above, at 1126 various clusters of data entitiesmay optionally be merged and/or collapsed when common data entitiesand/or properties are determined between the various clusters. Forexample, the system may determine that two different generated clustersboth include user 1158. Accordingly, the system may merge the twoclusters each including the common data entity (user 1158) into a singlecluster. Accordingly, in an embodiment the clustering method 920 b mayiteratively cluster related data entities.

In an embodiment, the various clustered data entities may includevarious properties and characteristics, including information regardingdata communications and requests between internal and external computersystems. For example, a given connection pair (or seed) may representmultiple connections over a period of time (as described above inreference to FIG. 11A). Accordingly, various information related to theconnections, including request sizes, may be included in the datacluster.

Additionally, in an embodiment a cluster graph similar to the clusterillustration of FIG. 11D may be made available to an analyst or otheruser of the data analysis system. For example, an analyst may select abutton (for example, an “Investigate in Graph” button) in a userinterface of the system to view a cluster graph of a selected cluster.

c. Beaconing Malware Detection: Cluster Scoring

Turning now to FIG. 11C, a flowchart of example cluster scoring methods930 b, 940 b of the data analysis system as applied to beaconing malwaredetection is shown, according to various embodiments of the presentdisclosure. The clustering scoring methods 930 b, 940 b may generally beunderstood to correspond to blocks 930 and 940 (cluster score andmetascore generation) of the generalized process of FIG. 9.Additionally, the clustering scoring methods 930 b, 940 b may correspondwith scoring strategies, as described above. In the flowchart of FIG.11C, block 1132 indicates that each of the following blocks (1134, 1136,1142, and 1144) may be performed for each of the clusters generated bythe cluster generation method 920 b of FIG. 11B.

At block 1134, the data analysis system may access and/or receivebeaconing scoring criteria. The beaconing scoring criteria may includeany number of rules or scoring strategies such that multiple scores maybe generated for each cluster. Several non-limiting examples ofbeaconing scoring criteria may include: a number of external domains inthe cluster known to be malicious; a number of blacklists on which anexternal domain in the cluster appears; a trustworthiness (and/ornumber) of blacklists on which external domains in the cluster appear; anumber and/or severity of host-based events in the cluster (such as, forexample, virus scan alerts and/or logged events, intrusion preventionsystem alerts and/or logged events, and the like); a number of requestsand/or connections between internal and external network devicesassociated with the cluster that were blocked by a proxy, router, orother appliance linking the internal network to the external network;and/or an average request size (for example, an amount of datatransmitted) between the internal and external devices associated withthe cluster (for example, smaller request sizes may indicate a higherlikelihood that the activity is related to beaconing activity).

At block 1136, the beaconing scoring criteria may be applied to theclusters and cluster scores may be generated. In an embodiment, eachcluster score may include an absolute value and/or a weighted value asdescribed above in reference to FIG. 10C. Additionally, as describedabove, the system may normalize the absolute values of each of thescores before applying a relative weighting to arrive at a weightedvalue. Examples of cluster scores presented to an analyst or other userof the data analysis system are shown and described below in referenceto FIG. 11E.

At block 1142, a metascore may be generated for the clusters. Thecluster metascore may be based on a combination or aggregation of theindividual scores generated in block 1136. Alternatively, the metascoresmay be separately determined scores. In an embodiment, a metascore maybe calculated by summing, multiplying, and/or otherwise aggregating oraveraging the various individual scores together. The metascore may, inan embodiment, capture the relative importance of each of the individualscores by weighting each of the individual scores in a manner similar tothat described above with reference to FIG. 10C. For example, as shown“known bad domains” may be weighted more heavily than other clusterscores as a known bad domain included in a cluster is a strong indicatorof malicious beaconing activity. In another example, “requests blockedby proxy” may be weighted less heavily than other cluster scores as ablocked proxy request may be an indicator of potentially maliciousbeaconing activity, but it may not be as strong an indicator as others.

In various embodiments, metascores and/or scores may advantageouslyenable an analyst to directly compare and/or prioritize variousclusters, and/or may advantageously be used by the data analysis systemto prioritize a list of clusters and/or scores related to a cluster.

At optional block 1144, analyst (or other user) feedback may optionallybe used in future scoring by the data analysis system. For example, ifthe analyst determines that a particular domain, identified by thesystem as potentially malicious, is not malicious, this information maybe used by the system in future scoring of clusters. For example, thedomain determined by the analyst to not be malicious may be whitelisted,or less weight may be applied to scores related to that domain.

d. Beaconing Malware Detection: Example User Interface

FIG. 11E illustrates an example cluster analysis user interface of thedata analysis system as applied to beaconing malware detection,according to an embodiment of the present disclosure. Similar to theexample user interface of FIG. 10E described above, the example userinterface of FIG. 11E includes a list of clusters 1182 (e.g., eachcluster may include multiple data entities associated with a particularseed connection pair), a list of scores 1184, and a detailed view of ascore 1186. In various embodiments, more or fewer elements may beincluded in the user interface, and/or the elements may be arrangeddifferently. Additionally, while certain aspects of the user interfaceof FIG. 11E may be similar to those of the user interface of FIG. 5described above, the user interface of the FIG. 11E includes a number ofdifferences. For example, differing from the user interface of FIG. 5,the user interface of the FIG. 11E may include a list of clusters in afirst column, a list of scores associated with a selected cluster in amiddle column, and/or details associated with a selected score in a lastcolumn. Such an arrangement may advantageously enable an analyst toinvestigate various scores associated with a cluster. Additionally,clusters in such an interface may advantageously be prioritizedaccording to any of multiple scores and/or metascores, as describedabove.

In the example user interface of FIG. 11E, an analyst or user hasselected “Beaconing Cluster 1.” Accordingly, various scores associatedwith that cluster may be displayed in the list of scores 1184. Forexample, scores are listed for “Known bad domain” and “Average requestsize,” among others. Additionally, in the example user interface, theanalyst has selected the “Average request size” score. Accordingly,details related to that score may be displayed in the detailed view1186.

According to an embodiment, various items of information may be includedin the user interface that may be useful to an analyst in evaluatingand/or investigating the generated clusters. For example, metascoresassociated with each of the generated clusters may be shown in the listof clusters 1182, and/or the clusters may be prioritized according tothe metascores. In another example, as described above with reference toFIG. 10E, absolute values and/or weighted values may be displayed in thelist of scores 1184 for each score. In another example, the detailedview 1186 may include a graph that shows additional information relatedto the selected score. For example, in FIG. 11E, the graph shown in thedetailed view 1186 shows a distribution of the request sizes associatedwith each connection to an external domain or IP address in the cluster.In the example, around 20 requests had a size around 1 megabyte, around100 requests had a size around 100 kilobytes, and around 1 request had asize around 1 kilobyte. In other embodiments, various other detailedinformation may be included in the user interface of FIG. 11E.

According to various embodiments, the data analysis system as applied tobeaconing malware detection may advantageously enable an analyst todetect and proactively remove an item of malware from various computersystems. Further, according to various embodiments the data analysissystem as applied to beaconing malware detection may advantageouslyenable an analyst to block particular domains determined to be relatedto beaconing malware, and/or take other step to protect and internalnetwork from attack.

In an embodiment, the data analysis system may automatically evaluatethe generated clusters to determine a likelihood that a given clusterrepresents beaconing malware activity. For example, the system maydetermine that a cluster having a metascore below a particular thresholdis likely not related to beaconing malware activity, while a clusterhaving a metascore above another particular threshold likely isbeaconing malware activity. In an embodiment, the system may determinethat a cluster having a metascore within a particular range ofthresholds requires additional analysis by an analyst as the likelihoodof beaconing malware activity is not conclusive. In an embodiment, ananalyst may adjust the thresholds, the metadata calculations, and/or theweighting applied to the scores. Further, the analyst may marks variousclusters as, for example, beaconing malware, likely beaconing malware,likely not beaconing malware, and/or not beaconing malware.Additionally, the analyst may dispatch other analysts to reviewparticular clusters and/or mark particular clusters for furtheranalysis.

XIV. Example Application of the Data Analysis System to MalwareUser-Agent Detection

FIGS. 12A-12E depict various aspects of the data analysis system asapplied to malware user-agent detection. For example, the data analysissystem may be used in conjunction with user-agent-related data entitiesto detect malware on a computer system or a network of computer systems.Malware activity may include, for example, a software programmaliciously installed on a target computer system that periodicallyattempts to transmit data and/or communicate with a remote computersystem. Typically, software programs, including malware softwareprograms, may include a user-agent string as part of a communicationwith another computer system. A user-agent string may include, forexample, various items of information regarding the source of thecommunication including, for example, an application type, an operatingsystem identifier, a software version, and/or the like. For example, abrowser application may include a user-agent string in a headerassociated with a network request, such as a Hypertext Transfer Protocol(HTTP) request. Such a browser user-agent string may include, forexample, a name and version of the browser, and a layout engine andlayout engine version of the browser. Examples of user-agent stringsassociated with particular browser applications may include “Mozilla/4.0(Windows; MSIE 6.0; Windows NT 6.0)”, “Mozilla/5.0 (Windows; U; MSIE9.0; Windows NT 9.0; en-US))”, and the like. Other software applicationsmay also include user-agent strings along with network communications.As user-agent strings include information related to the requestingsoftware application, they may be useful in detecting malicioussoftware, or malware. For example, unusual user-agent strings may bedetected by the data analysis system and may be flagged as potentiallymalicious. In another example, a whitelist of common user-agent stringsmay be used to filter out particular known-good networks requests, anddetermine other network requests that may be associated with malware.

According to various embodiments, user-agent-related data entity seedsmay be generated by the system as described below in reference to FIG.12A. Each of these user-agent-related entity seeds may include adetected user-agent string entity, and the seeds may be generated basedon a likelihood that the seeds represent malware-related activitiesand/or data. Each of the seeds may be used as a basis for clusteringvarious other user-agent-related entities, as described in reference toFIGS. 12B and 12D. Accordingly, the generated clusters may representvarious data entities that are all related to potential malware-relatedactivity. Each of the generated clusters may then be scored according tovarious criteria (or rules), as described below in reference to FIG.12C. The various scores and metascores generated by the system mayprovide indications to an analyst regarding the likelihood that thecluster includes entities representing malware-related activities and/ordata. The information, including the clusters and scores generated bythe data analysis system, may be presented to an analyst via a userinterface as described below in reference to FIG. 12E. Advantageously,according to an embodiment, the analyst may sort the clusters accordingto their determined scores (and/or metascores) so as to prioritizeinvestigations into potential malware. Further, the data analysis systemmay advantageously automatically cluster or group many related dataentities to enable rapid investigation and evaluation by an analyst todetect likely malware.

In an embodiment, and as described below, the data analysis system maybe used in a network environment in which an internal network is incommunication with an external network. The system may be used todetermine, for example, whether any computer systems of the internalnetwork have been infected by malware that is communicating withcomputer systems of the external network. Various computerized devicesmay be included in the internal network that may be capable to capturingand/or logging data traffic between the internal network and theexternal network including, for example, network routers and/orswitches.

a. Malware User-Agent Detection: Seed Generation

FIG. 12A is a flowchart of an example of a seed generation method 910 cof the data analysis system as applied to malware user-agent detection,according to various embodiments of the present disclosure. The seedgeneration method 910 c may generally be understood to correspond toblock 910 (seed generation) of the generalized process of FIG. 9.

Referring to FIG. 12A, at block 1212, network communications and/or datatraffic information, including user-agent strings, between the internaland external networks may be captured by the data analysis system.Various items of information may be captured including, for example,external IP addresses contacted, external domains contacted, internalcomputer systems from which data is transmitted to external devices, andthe like. These items of information may be captured by, for example, anetwork traffic router that connects the internal and external networksto one another. The network traffic router may, for example, log suchitems of information such that they may be read and analyzed by the dataanalysis system. Alternatively, the network traffic may be captured by,for example, other types of computerized sensors. Each of the abovedescribed items of information may be a data entity in the context ofthe data analysis system.

At block 1213, the system may filter out particular user-agent strings,and associated data entities, based on various criteria so as to removeany communications that are not likely to be malicious. For example, thesystem may make use of a list of whitelisted domains (1213 a) byremoving any communications items that involve communications with knownnon-malicious domains. Similarly, the system may make use of a list ofwhitelisted user-agent strings by removing any communications items thatinvolve communications with known non-malicious user-agent strings. Inanother example, the system may determine that particular connectionsbetween certain internal computer systems and external computer systemsare common or frequent (1213 b), and thus not likely malicious. In anembodiment, common connections between many internal computer systemsand particular external computer systems may indicate that theconnections are not performed by malicious software. Such commonconnection may be associated with, for example, software updates bylegitimate software installed on many computer systems of the internalnetwork. In yet another example, the data analysis system may optionallyfilter out any communications related to random user-agent strings (1213c).

Once a set of communications associated with the filtered list ofuser-agent strings is determined, at block 1214 the data analysis systemcompares the detected, filtered, user-agent strings from a test periodto the detected, filtered, user-agent strings from a reference period.In an embodiment, a test period may include a current day, while areference period may include a three, four, or five week period beforethe current day. Any other test and/or reference period may be used bythe data analysis system. According to an embodiment, comparinguser-agent strings between a test period and a reference period enablesthe data analysis system to detect any new (for example, not previouslyused during the reference period) user-agent strings. At block 1215, anyuser-agent strings determined to be new (for example, used during thetest period but not during the reference period) may be used (along withrelated communication data entities) as seeds.

At block 1216, the data analysis system may use an additional oralternative method for determining seeds. This may be accomplished bydetermining a frequency distribution of detected, filtered, user-agentstrings used by hosts (or various computer systems) of the internalnetwork. Such a frequency distribution may be determined for aparticular test period of time. Additionally, such a frequencydistribution may indicate, for example, the number of hosts that usedany particular user-agent string during the test period of time. Asindicated by block 1218, any user-agent strings that are determined tohave been used rarely (for example, used by less than a particularthreshold number or percentage of hosts) may be used (along with relatedcommunication data entities) as seeds by the data analysis system. Forexample, a particular threshold number of hosts may be five. In thisexample, the system may determine that a particular user-agent was usedby two hosts during the test period of time. Accordingly, the system maydesignate the particular user-agent as a seed. Alternatively, the systemmay determine that a second particular user-agent was used by 100 hostsduring the test period of time. Accordingly, the system may notdesignate the second particular user-agent as a seed.

b. Malware User-Agent Detection: Cluster Generation

Turning now to FIG. 12B, a flowchart of an example of a clusteringmethod 920 c of the data analysis system as applied to malwareuser-agent detection is shown, according to various embodiments of thepresent disclosure. The clustering method 920 c may generally beunderstood to correspond to block 920 (cluster generation) of thegeneralized process of FIG. 9. Additionally, the clustering method 920 cmay correspond to a clustering strategy, as described above. In theflowchart of FIG. 12B, block 1222 indicates that the following block(1224) may be performed for each of the seeds generated by the seedgeneration method 910 c of FIG. 12A.

At block 1224, any user-agent-related data entities that are related tothe seed may be clustered. Clustering of data entities may beaccomplished as generally described above, in which data bindings areexecuted and/or searching and filtering are performed (through, forexample, a generic interface to various data sources) as part of aclustering strategy. Additionally, as described above, clustered dataentities may be related by, for example, sharing the same or similarproperties, characteristics, and/or metadata. Examples of data entitiesthat may be clustered include, but are not limited to: hosts or sourcecomputing systems (for example, computing systems of the internalnetwork); host owners, such as users of hosts or computing systems (forexample, persons having accounts on particular computer systems);host-based events (such as, for example, virus scan alerts and/or loggedevents, intrusion prevention system alerts and/or logged events, and thelike); internal IP addresses; internal IP addresses that connect toexternal domains; internal computer systems; internal computer systemsthat connect to external domains; external IP addresses; externaldomains; external IP addresses associated with external domains; and thelike.

FIG. 12D illustrates an example growth of a cluster of related dataentities in a malware user-agent detection application, according to anembodiment of the present disclosure. In FIG. 12D, boxes indicate dataentities, while lines between boxes indicate links that connect dataentities. As described above, seeds in the described malware user-agentdetection application of the data analysis system may be user agents. Asshown in the example of FIG. 12D, a seed user agent 1252 has beengenerated (such as by the process of FIG. 12A) as indicated visually bythe internal seed dashed line. Then, in a clustering step correspondingto block 1224 (of FIG. 12B) and represented by the external clusterdashed line 1254, various other data entities related to the seed entitymay be added to the cluster. For example, the data analysis system hasclustered two hosts, 1256 and 1258, that transmitted data including theuser agent 1252.

Returning again to FIG. 12B, dashed line 1226 indicates that the clustergeneration method may optionally repeat multiple times until, forexample, the clustering strategy is completed and/or no additionalrelated data entities are found by the system. For example, in referenceto FIG. 12D, additional data entities may be clustered including user1262 and host-based event 1264 (each related to the host 1256), and user1266 and external domain 1268 (each related to the host 1258). Asindicated by ellipses 1260, 1272, and 1274, additional data entities maybe clustered in each or subsequent clustering steps. Further, referringto FIG. 12B, and as described above, at 1226 various clusters of dataentities may optionally be merged and/or collapsed when common dataentities and/or properties are determined between the various clusters.For example, the system may determine that two different generatedclusters both include host 1258. Accordingly, the system may merge thetwo clusters each including the common data entity (host 1258) into asingle cluster. Accordingly, in an embodiment the clustering method 920c may iteratively cluster related data entities.

In an embodiment, the various clustered data entities may includevarious properties and characteristics, including information regardingdata communications and requests between internal and external computersystems and the associated user-agent strings. For example, a givenconnection between a host and an external domain may represent multipleconnections over a period of time. In an embodiment, the data analysissystem may cluster various data entities with the seed so as to enablean analyst to quickly determine, for example, any computer systemsassociated with the user agent, any users of those computer systems, anyactions (including communications) taken by those computer systems, anyevents happening on those computer systems (for example, host-basedevents, including file changes, software installs, and the like), andthe like.

Additionally, in an embodiment a cluster graph similar to the clusterillustration of FIG. 12D may be made available to an analyst or otheruser of the data analysis system. For example, an analyst may select abutton (for example, an “Investigate in Graph” button) in a userinterface of the system to view a cluster graph of a selected cluster.

c. Malware User-Agent Detection: Cluster Scoring

Turning now to FIG. 12C, a flowchart of example cluster scoring methods930 c, 940 c of the data analysis system as applied to malwareuser-agent detection is shown, according to various embodiments of thepresent disclosure. The clustering scoring methods 930 c, 940 c maygenerally be understood to correspond to blocks 930 and 940 (clusterscore and metascore generation) of the generalized process of FIG. 9.Additionally, the clustering scoring methods 930 c, 940 c may correspondwith scoring strategies, as described above. In the flowchart of FIG.12C, block 1232 indicates that each of the following blocks (1234, 1236,1242, and 1244) may be performed for each of the clusters generated bythe cluster generation method 920 c of FIG. 12B.

At block 1234, the data analysis system may access and/or receiveuser-agent scoring criteria. The user-agent scoring criteria may includeany number of rules or scoring strategies such that multiple scores maybe generated for each cluster. Several non-limiting examples ofuser-agent scoring criteria may include: a number of external domains inthe cluster known to be malicious; a number of user agents known to bemalicious (including, for example, being included on a blacklist or anumber of blacklists); a number of blacklists on which an externaldomain or user agent in the cluster appears; a trustworthiness (ornumber) of blacklists on which external domains or user agents in thecluster appear; a number and/or severity of host-based events in thecluster (such as, for example, virus scan alerts and/or logged events,intrusion prevention system alerts and/or logged events, and the like);a number of requests and/or connections between internal and externalnetwork devices associated with the cluster that were blocked by aproxy, router, or other appliance linking the internal network to theexternal network; and/or an average request size (for example, an amountof data transmitted) between internal and external devices associatedwith the cluster (for example, smaller request sizes may indicate ahigher likelihood that the activity is related to malware activity).

At block 1236, the user-agent scoring criteria may be applied to theclusters and cluster scores may be generated. In an embodiment, eachcluster score may include an absolute value and/or a weighted value asdescribed above in reference to FIG. 10C. Additionally, as describedabove, the system may normalize the absolute values of each of thescores before applying a relative weighting to arrive at a weightedvalue. Examples of cluster scores presented to an analyst or other userof the data analysis system are shown and described below in referenceto FIG. 12E.

At optional block 1242, a metascore may optionally be generated for theclusters. The cluster metascore may be based on a combination oraggregation of the individual scores generated in block 1236.Alternatively, the metascores may be separately determined scores. In anembodiment, a metascore may be calculated by summing, multiplying,and/or otherwise aggregating or averaging the various individual scorestogether. The metascore may, in an embodiment, capture the relativeimportance of each of the individual scores by weighting each of theindividual scores in a manner similar to that described above withreference to FIG. 10C.

In various embodiments, metascores and/or scores may advantageouslyenable an analyst to directly compare and/or prioritize variousclusters, and/or may advantageously be used by the data analysis systemto prioritize a list of clusters and/or scores related to a cluster.

At optional block 1244, an analyst (or other user) may optionallyinvestigate particular hosts included in a cluster and/or associatedwith a particular user agent. The analyst may further optionally place auser agent determined to be associated with malicious software on ablacklist that may be used in subsequent applications of the dataanalysis system.

d. Malware User-Agent Detection: Example User Interface

FIG. 12E illustrates an example cluster analysis user interface of thedata analysis system as applied to malware user-agent detection,according to an embodiment of the present disclosure. Similar to theexample user interface of FIG. 11E described above, the example userinterface of FIG. 12E includes a list of clusters 1282, a list of scores1284, and a detailed view of a score 1286. In various embodiments, moreor fewer elements may be included in the user interface, and/or theelements may be arranged differently. Additionally, while certainaspects of the user interface of FIG. 12E may be similar to those of theuser interface of FIG. 5 described above, the user interface of the FIG.12E includes a number of differences. For example, differing from theuser interface of FIG. 5, the user interface of the FIG. 12E may includea list of clusters in a first column, a list of scores associated with aselected cluster in a middle column, and/or details associated with aselected score in a last column. Such an arrangement may advantageouslyenable an analyst to investigate various scores associated with acluster. Additionally, clusters in such an interface may advantageouslybe prioritized according to any of multiple scores and/or metascores, asdescribed above.

In the example user interface of FIG. 12E, an analyst or user hasselected “User-agent Cluster 1.” Accordingly, various scores associatedwith that cluster may be displayed in the list of scores 1284. Forexample, scores are listed for “Blacklisted user agents” and “Averagerequest size,” among others. Additionally, in the example userinterface, the analyst has selected the “Average request size” score.Accordingly, details related to that score may be displayed in thedetailed view 1286.

According to an embodiment, various items of information may be includedin the user interface that may be useful to an analyst in evaluatingand/or investigating the generated clusters. For example, metascoresassociated with each of the generated clusters may be shown in the listof clusters 1282, and/or the clusters may be prioritized according tothe metascores. In another example, as described above with reference toFIG. 12E, absolute values and/or weighted values may be displayed in thelist of scores 1284 for each score. In another example, the detailedview 1286 may include a graph that shows additional information relatedto the selected score. For example, in FIG. 12E, the graph shown in thedetailed view 1286 shows a distribution of the request sizes associatedwith each connection to an external domain or IP address in the cluster,as described above in reference to FIG. 11E.

According to various embodiments, the data analysis system as applied tomalware user-agent detection may advantageously enable an analyst todetect and proactively remove an item of malware from various computersystems. Further, according to various embodiments the data analysissystem as applied to malware user-agent detection may advantageouslyenable an analyst to block particular domains determined to be relatedto malicious software, and/or take other step to protect and internalnetwork from attack.

In an embodiment, the data analysis system may automatically evaluatethe generated clusters to determine a likelihood that a given clusterrepresents malicious activity. For example, the system may determinethat a cluster having a metascore below a particular threshold is likelynot related to malware activity, while a cluster having a metascoreabove another particular threshold likely is malware activity. In anembodiment, the system may determine that a cluster having a metascorewithin a particular range of thresholds requires additional analysis byan analyst as the likelihood of malware activity is not conclusive. Inan embodiment, an analyst may adjust the thresholds, the metadatacalculations, and/or the weighting applied to the scores. Further, theanalyst may mark various clusters as, for example, malware, likelymalware, likely not malware, and/or not malware. Additionally, theanalyst may dispatch other analysts to review particular clusters and/ormark particular clusters for further analysis.

XV. Example Application of the Data Analysis System to Activity TrendDetection

FIGS. 13A-13E depict various aspects of the data analysis system asapplied to activity trend detection. For example, the data analysissystem may be used in conjunction with activity-related data entities todetect activity trends and/or behavior trends on a computer system (alsoreferred to as a computerized device) or a network of computer systems(or computerized devices). Examples of activities and/or behaviors thatmay be detected by the data analysis system include host-based events(for example, events occurring on a computer system and/or computerizeddevice) such as warning and/or alerts provided by an anti-virus softwareprogram and/or an intrusion detection software program; file changes ona computer system (for example, file modifications, file deletions, filepermissions changes, and the like); file accesses on a computer system(for example, accesses by a user of a computer system); and/or changesin processes and/or software applications running on a computer system(for example, spawning and/or killing of processes); among others.According to various embodiments, the data analysis system may be usedto profile activity and/or behaviors associated with a computer systemso as to detect trends. For example, the data analysis system may detectan unusual increase in a particular type of activity that an analyst maythen investigate. In a specific example, the system may help an analystdetect a user accessing a large number of files and modifying the filecontents, possibly maliciously. In another example, a previouslyundetected malicious software program may modify other software programsrunning on a computer system in ways that are unusual, and the dataanalysis system may detect such modifications.

According to various embodiments, activity-related data entity seeds maybe generated by the system as described below in reference to FIG. 13A.Each of these activity-related entity seeds may include a detectedactivity-related entity, such as a host-based event, and the seeds maybe generated based on a likelihood that the seeds represent trends inactivities and/or behavior. Each of the seeds may be used as a basis forclustering various other activity-related entities, as described inreference to FIGS. 13B and 13D. Accordingly, the generated clusters mayrepresent various data entities that are all related to potentiallyinteresting activity trends. Each of the generated clusters may then bescored according to various criteria (or rules), as described below inreference to FIG. 13C. The various scores generated by the system mayprovide indications to an analyst regarding the likelihood that thecluster includes entities representing particular activity trends. Theinformation, including the clusters and scores generated by the dataanalysis system, may be presented to an analyst via a user interface asdescribed below in reference to FIG. 13E. Advantageously, according toan embodiment, the analyst may sort the clusters according to theirdetermined scores so as to prioritize investigations into potentiallyinterested activity trends. Further, the data analysis system mayadvantageously automatically cluster or group many related data entitiesto enable rapid investigation and evaluation by an analyst to detectactivity trends.

In an embodiment, and as described below, the data analysis system maybe used in a network environment in which an internal network is incommunication with an external network. The internal network may includemultiple computer systems (also referred to as hosts) that may be incommunication with one another. Various activities and events occurringon each of the hosts, as well as network communications, may be trackedand/or logged by aspects of the data analysis system such that they maybe analyzed as described below. Various computerized devices may beincluded in the network that may be capable to capturing and/or loggingdata traffic and communications, for example, network routers and/orswitches. Further, various software programs or other aspects may beincluded in the network and/or on computer systems in the network thatmay be capable to capturing and/or logging various host-based events.

a. Activity Trend Detection: Seed Generation

FIG. 13A is a flowchart of an example of a seed generation method 910 dof the data analysis system as applied to activity trend detection,according to various embodiments of the present disclosure. The seedgeneration method 910 d may generally be understood to correspond toblock 910 (seed generation) of the generalized process of FIG. 9.

Referring to FIG. 13A, at block 1312, various host-based events may becaptured and/or received by the data analysis system. These events maybe stored in logs (1312 a) on individual computer systems and thenretrieved by the data analysis system, or may be provided to the dataanalysis system as they are captured. Additionally, the events may eachbe a data entity. Various types of host-based events may be capturedincluding, for example (and as mentioned above), warning and/or alertsprovided by an anti-virus software program, other monitoring software(1312 b) and/or an intrusion detection software program (1312 c); filechanges on a computer system (for example, file modifications, filedeletions, file permissions changes, and the like); file accesses on acomputer system (for example, accesses by a user of a computer system);and/or changes in processes and/or software applications running on acomputer system (for example, spawning and/or killing of processes);among others. These host-based events and other items of information maybe captured by, for example, specialized network devices and/or softwareapplications running on various computer systems in the network. In anembodiment, the data analysis system may further capture network trafficinformation, as described above in references to FIGS. 11A and 12A.

As various types of events may be captured by the data analysis system,in the flowchart of FIG. 13A, block 1313 indicates that each of thefollowing blocks (1314, 1315, and 1316) may be performed for each typeof event that is captured in block 1312. In an embodiment, the capturedevents may be grouped according to type, such that each type-group maybe processed according to blocks 1314, 1315, and 1316.

At block 1314, for a particular type of captured event on a particularhost, any events of the particular type captured on the particular hostmay be analyzed by the data analysis system and a current Z-score may becalculated with respect to a previous time period. For example, in anembodiment the data analysis system may determine any of a particulartype of event that is captured on a particular host. The data analysissystem may also determine a current time period and a previous timeperiod into which the events of a particular type may be divided. In oneexample, a current time period may be a most recent day, week, or month.In an example, a previous time period may be a day, week, or monthimmediately preceding the current time period. Then, the data analysissystem may calculate a probability distribution of events of theparticular type on the host during the previous time period. Next, thedata analysis system may determine a Z-score of events of the particulartype on the host during the current time period along the previous timeperiod distribution. A Z-score may indicate, for example, a number ofstandard deviations the events during the current time period are aboveor below the mean of the distribution of the previous time period. In anembodiment, the calculated Z-score may provide an indication of anunusual change in activity of a particular type of event on a particularhost.

At block 1315, the data analysis system may compare the calculatedZ-score, for each event type on each host, to a threshold. The thresholdmay be, for example, +/−1, +/−1.5, +/−2, +/−2.5, among otherpossibilities. At block 1316, any events having Z-scores that satisfythe threshold may be used as seeds. Accordingly, the method 910 d maygenerate seeds that may be used by the data analysis system in anactivity trend detection application.

b. Activity Trend Detection: Cluster Generation

Turning now to FIG. 13B, a flowchart of an example of a clusteringmethod 920 d of the data analysis system as applied to activity trenddetection is shown, according to various embodiments of the presentdisclosure. The clustering method 920 d may generally be understood tocorrespond to block 920 (cluster generation) of the generalized processof FIG. 9. Additionally, the clustering method 920 d may correspond to aclustering strategy, as described above. In the flowchart of FIG. 13B,block 1322 indicates that the following block (1324) may be performedfor each of the seeds generated by the seed generation method 910 d ofFIG. 13A.

At block 1324, any activity-related data entities that are related tothe seed may be clustered. Clustering of data entities may beaccomplished as generally described above, in which data bindings areexecuted and/or searching and filtering are performed (through, forexample, a generic interface to various data sources) as part of aclustering strategy. Additionally, as described above, clustered dataentities may be related by, for example, sharing the same or similarproperties, characteristics, and/or metadata. Examples of data entitiesthat may be clustered include, but are not limited to: hosts or sourcecomputing systems (for example, computing systems of the network); hostowners, such as users of hosts or computing systems (for example,persons having accounts on particular computer systems); host-basedevents related to the seed event (such as, for example, virus scanalerts and/or logged events, intrusion prevention system alerts and/orlogged events, and the like); internal IP addresses; internal IPaddresses that connect to external domains; internal computer systems;internal computer systems that connect to external domains; external IPaddresses; external domains; external IP addresses associated withexternal domains; and the like.

FIG. 13D illustrates an example growth of a cluster of related dataentities in a malware activity trend detection application, according toan embodiment of the present disclosure. In FIG. 13D, boxes indicatedata entities, while lines between boxes indicate links that connectdata entities. As described above, seeds in the described activity trenddetection application of the data analysis system may be host-basedevents. As shown in the example of FIG. 13D, a seed host-based event1352 (for example, a spawned process on a particular host, where thenumber “85674” may indicate, for example, a process identifier) has beengenerated (such as by the process of FIG. 13A) as indicated visually bythe internal seed dashed line. Then, in a clustering step correspondingto block 1324 (of FIG. 13B) and represented by the external clusterdashed line 1354, various other data entities related to the seed entitymay be added to the cluster. For example, the data analysis system hasclustered a host 1356 (on which the event 1352 occurred), and anotherrelated host-based event 1358 (that may, for example, havecharacteristics similar to those of event 1352).

Returning again to FIG. 13B, dashed line 1326 indicates that the clustergeneration method may optionally repeat multiple times until, forexample, the clustering strategy is completed and/or no additionalrelated data entities are found by the system. For example, in referenceto FIG. 13D, additional data entities may be clustered including user1360, host-based event 1362, and external domain 1364 (each related tothe host 1356), and host 1368 and user 1370 (each related to the event1358). As indicated by ellipses 1366, additional data entities may beclustered in each or subsequent clustering steps. Further, referring toFIG. 13B, and as described above, at 1326 various clusters of dataentities may optionally be merged and/or collapsed when common dataentities and/or properties are determined between the various clusters.For example, the system may determine that two different generatedclusters both include event 1358. Accordingly, the system may merge thetwo clusters each including the common data entity (event 1358) into asingle cluster. Accordingly, in an embodiment the clustering method 920d may iteratively cluster related data entities.

In an embodiment, the various clustered data entities may includevarious properties and characteristics, including information regardingdata communications and requests between computer systems in thenetwork. For example, a given connection between as host and an externaldomain may represent multiple connections over a period of time. In anembodiment, the data analysis system may cluster various data entitieswith the seed so as to enable an analyst to quickly determine, forexample, any computer systems and/or other events associated with aparticular detected unusual event, any users of related computersystems, any actions (including communications) taken by relatedcomputer systems, any events happening on related computer systems (forexample, host-based events, including file changes, software installs,and the like), and the like.

Additionally, in an embodiment a cluster graph similar to the clusterillustration of FIG. 13D may be made available to an analyst or otheruser of the data analysis system. For example, an analyst may select abutton (for example, an “Investigate in Graph” button) in a userinterface of the system to view a cluster graph of a selected cluster.

c. Activity Trend Detection: Cluster Scoring

Turning now to FIG. 13C, a flowchart of an example cluster scoringmethod 930 d of the data analysis system as applied to malware activitytrend detection is shown, according to various embodiments of thepresent disclosure. The clustering scoring method 930 d may generally beunderstood to correspond to block 930 (cluster score generation) of thegeneralized process of FIG. 9. Additionally, the clustering scoringmethod 930 d may correspond with scoring strategies, as described above.In the flowchart of FIG. 13C, block 1332 indicates that each of thefollowing blocks (1333, 1334, 1335, and 1336) may be performed for eachof the clusters generated by the cluster generation method 920 d of FIG.13B.

At block 1334, the data analysis system may access and/or received thepreviously determined Z-score of the seed. This Z-score may beconsidered by the data analysis system as a Z-score of the cluster, asit indicates how the detected event (the seed) deviates from the normalactivity on the host. At block 1335, the cluster Z-score may be comparedto a calculated “universe,” or entire-network, Z-score. In anembodiment, the universe Z-score may be calculated in a manner similarto the calculation of the seed Z-score, but with reference to the entirenetwork of computers from which the data analysis system has capturedevents. For example, the universe Z-score may be calculated as follows:The data analysis system may use the same current and previous timeperiods as were used in calculation of the seed Z-score, however aprobability distribution of events (of the same or similar type to theseed event) across the network may be calculated for the previous timeperiod. Then, the data analysis system may determine a universe Z-scoreof events of the particular type on the network during the current timeperiod along the previous time period distribution. In an embodiment,the calculated universe Z-score may provide an indication of an unusualchange in activity of a particular type of event on the network.

In various embodiments, the comparison between the cluster Z-score andthe universe Z-score may provide an indication, for example, of whetheror not the activity trend detected on the host is unusual in the contextof the network. For example, when the activity trend detected on thehost is also present on the network as a whole (for example, the clusterZ-score is similar to the universe Z-score), the activity trend may notbe particularly interesting to an analyst. Alternatively, when theactivity trend detected on the host is different from related activitythat is present on the network as a whole (for example, the clusterZ-score is different from the universe Z-score), the activity trend maybe interesting to an analyst. Accordingly, at block 1336, a clusterscore is generated that indicates how significantly the cluster Z-scoreand the universe Z-score differ from one another. Additionally, in anembodiment, the cluster score may capture the absolute value of thecluster Z-score such that larger cluster Z-scores (indicatingsignificant activity trends) may result in larger cluster scores ascompared to smaller cluster scores, even in view of the differencebetween the cluster Z-score and the universe Z-score.

d. Activity Trend Detection: Example User Interface

FIG. 13E illustrates an example cluster analysis user interface of thedata analysis system as applied to activity trend detection, accordingto an embodiment of the present disclosure. Similar to the example userinterface of FIG. 11E described above, the example user interface ofFIG. 13E includes a list of clusters 1382, a list of scores 1384, and adetailed view of score information associated with the cluster 1386. Invarious embodiments, more or fewer elements may be included in the userinterface, and/or the elements may be arranged differently.Additionally, while certain aspects of the user interface of FIG. 13Emay be similar to those of the user interface of FIG. 5 described above,the user interface of the FIG. 13E includes a number of differences. Forexample, differing from the user interface of FIG. 5, the user interfaceof the FIG. 13E may include a list of clusters in a first column, a listof scores associated with a selected cluster in a middle column, and/ordetails associated with a selected score in a last column. Such anarrangement may advantageously enable an analyst to investigate variousscores associated with a cluster. Additionally, clusters in such aninterface may advantageously be prioritized according to any of multiplescores, as described above.

In the example user interface of FIG. 13E, an analyst or user hasselected “Activity Trend Cluster 1.” Accordingly, various scoresassociated with that cluster may be displayed in the list of scores1384. For example, a cluster Z-score, a universe Z-score, and a clusterscore 1390 may be listed. As described above the cluster score may becalculated based on the cluster Z-score and the universe Z-score.Additionally, in the example user interface, the analyst has selectedthe “Cluster Score.” Accordingly, details related to the cluster scoremay be displayed in the detailed view 1386.

According to an embodiment, various items of information may be includedin the user interface that may be useful to an analyst in evaluatingand/or investigating the generated clusters. For example, the clustersmay be prioritized according to the cluster scores. In another example,the detailed view 1386 may include a graph that shows additionalinformation related to the selected score. For example, in FIG. 13E, thegraph shown in the detailed view 1386 shows a graph with a probabilitydistribution that may be associated a universe of activity, activity ona particular computer, activity during a particular time period, and/orthe like. In an embodiment, the detailed view 1386 may include a graphthat may show Z-score values associated with the cluster.

In various embodiments, any other mathematical and/or statisticalmeasures and/or methods may be used in determining and/or detectingunusual and/or interesting activity trends. For example, any othermathematical methods may be used to determine activities that arestatistical outliers. Additionally, in various embodiments, the othermethods may be used in any of the seed generation, cluster generation,and/or scoring of clusters.

According to various embodiments, the data analysis system as applied toactivity trend detection may advantageously enable an analyst to detectand proactively take action when unusual activity is detected. In anembodiment, the data analysis system may automatically evaluate thegenerated clusters to determine a likelihood that a given clusterrepresents an activity trend or unusual activity. For example, thesystem may determine that a cluster having a cluster score below aparticular threshold is likely not an activity trend, while a clusterhaving a cluster score above another particular threshold likely is anactivity trend. In an embodiment, the system may determine that acluster having a cluster score within a particular range of thresholdsrequires additional analysis by an analyst as the likelihood of anactivity trend is not conclusive. In an embodiment, an analyst mayadjust the thresholds, the cluster score calculations, and/or theweighting applied to the Z-scores. Further, the analyst may mark variousclusters as, for example, activity trend, likely activity trend, likelynot activity trend, and/or not activity trend. Additionally, the analystmay dispatch other analysts to review particular clusters and/or markparticular clusters for further analysis.

XVI. Additional Embodiments

While the foregoing is directed to various embodiments, other andfurther embodiments may be devised without departing from the basicscope thereof. For example, aspects of the present disclosure may beimplemented in hardware or software or in a combination of hardware andsoftware. An embodiment of the disclosure may be implemented as aprogram product for use with a computer system. The program(s) of theprogram product define functions of the embodiments (including themethods described herein) and may be contained on a variety ofcomputer-readable storage media. Illustrative computer-readable storagemedia include, but are not limited to: (i) non-writable storage media(e.g., read-only memory devices within a computer such as CD-ROM disksreadable by a CD-ROM drive, flash memory, ROM chips or any type ofsolid-state non-volatile semiconductor memory) on which information ispermanently stored; and (ii) writable storage media (e.g., hard-diskdrive or any type of solid-state random-access semiconductor memory) onwhich alterable information is stored. Each of the processes, methods,and algorithms described in the preceding sections may be embodied in,and fully or partially automated by, code modules executed by one ormore computer systems or computer processors comprising computerhardware. The processes and algorithms may alternatively be implementedpartially or wholly in application-specific circuitry.

The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure. In addition, certain method or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto can be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically disclosed, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe disclosed example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the disclosed example embodiments.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

The term “comprising” as used herein should be given an inclusive ratherthan exclusive interpretation. For example, a general purpose computercomprising one or more processors should not be interpreted as excludingother computer components, and may possibly include such components asmemory, input/output devices, and/or network interfaces, among others.

Any process descriptions, elements, or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved, as would be understood by those skilled in the art.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure. The foregoing description details certainembodiments of the invention. It will be appreciated, however, that nomatter how detailed the foregoing appears in text, the invention may bepracticed in many ways. As is also stated above, it should be noted thatthe use of particular terminology when describing certain features oraspects of the invention should not be taken to imply that theterminology is being re-defined herein to be restricted to including anyspecific characteristics of the features or aspects of the inventionwith which that terminology is associated. The scope of the inventionshould therefore be construed in accordance with the appended claims andany equivalents thereof.

1. (canceled)
 2. A computer-implemented method comprising: generating,based on a plurality of captured communications, a filtered collectionof captured communications by selecting captured communications thatinclude a user-agent string and removing captured communications withdestinations on an approved list of destinations, wherein the approvedlist of destinations indicate destinations that are unlikely to berelated to malware activity; determining, based on the filteredcollection of captured communications, a first set of capturedcommunications associated with a test time period and a second set ofcaptured communications associated with a reference time period;identifying a first captured communication in the first set that is notincluded among the second set of captured communications, wherein thefirst captured communication indicates a new user-agent string notpreviously associated with the reference time period; and designatingthe new user-agent string as a seed; and generating a data item clusterbased on the seed, wherein generating the data item cluster comprises:adding the seed to the data item cluster; and adding to the data itemcluster one or more user-agent-related data items determined to beassociated with the seed, wherein the one or more user-agent-relateddata items comprises information associated with a computing device. 3.The computer-implemented method of claim 2, wherein the one or moreuser-agent-related data items further include at least one of: a user ofa particular computing device, an internal Internet Protocol address, anexternal Internet Protocol address, an external domain, an internalcomputing device, an external computing device, or a host-based event.4. The computer-implemented method of claim 3, further comprising:identifying the one or more user-agent-related data items based at leaston a clustering strategy, wherein the clustering strategy queries one ormore cluster data sources to determine at least one of: originating hostor destination computing devices associated with the seed, users oforiginating host computing devices, intrusion prevention system alertsassociated with originating host computing devices, internal InternetProtocol addresses associated with originating host computing devices,external Internet Protocol addresses associated with destinationcomputing devices, or external domains associated with the firstcaptured communication.
 5. The computer-implemented method of claim 2,further comprising: determining a score for the data item cluster; andcausing presentation of the data item cluster and the score in a userinterface of a client computing device.
 6. The computer-implementedmethod of claim 2, wherein generating, based on the plurality ofcaptured communications, the filtered collection of capturedcommunications further includes removing captured communications withrespective user-agent strings on an approved list of user-agent strings,wherein the approved list of user-agent strings indicate communicationsthat are unlikely to be related to malware activity.
 7. Thecomputer-implemented method of claim 2, wherein generating, based on theplurality of captured communications, the filtered collection ofcaptured communications further includes removing capturedcommunications associated with a particular external computer system,wherein the particular external computer system is unlikely to berelated to malware activity.
 8. The computer-implemented method of claim2, further comprising: identifying a quantity of appearances of the newuser-agent string in corresponding captured communications among thefirst set of captured communications; and determining the quantity isbelow a predetermined threshold.
 9. A non-transitory computer-readablestorage medium storing computer-executable instructions that, whenexecuted by a computer system, configure the computer system to performoperations comprising: generating, based on a plurality of capturedcommunications, a filtered collection of captured communications byselecting captured communications that include a user-agent string andremoving captured communications with respective user-agent strings onan approved list of user-agent strings, wherein the approved list ofdestinations indicate destinations that are unlikely to be related tomalware activity; determining, based on the filtered collection ofcaptured communications, a first set of captured communicationsassociated with a test time period and a second set of capturedcommunications associated with a reference time period; identifying afirst captured communication in the first set that is not included amongthe second set of captured communications, wherein the first capturedcommunication indicates a new user-agent string not previouslyassociated with the reference time period; and designating the newuser-agent string as a seed; and generating a data item cluster based onthe seed, wherein generating the data item cluster comprises: adding theseed to the data item cluster; and adding to the data item cluster oneor more user-agent-related data items determined to be associated withthe seed, wherein the one or more user-agent-related data itemscomprises information associated with a computing device.
 10. Thenon-transitory computer-readable storage medium of claim 9, wherein theone or more user-agent-related data items further include at least oneof: a user of a particular computing device, an internal InternetProtocol address, an external Internet Protocol address, an externaldomain, an internal computing device, an external computing device, or ahost-based event.
 11. The non-transitory computer-readable storagemedium of claim 9, wherein the computer-executable instructions furtherconfigure the computer system to perform operations comprising:determining a score for the data item cluster; and causing presentationof the data item cluster and the score in a user interface of a clientcomputing device.
 12. The non-transitory computer-readable storagemedium of claim 9, wherein the computer-executable instructions furtherconfigure the computer system to perform operations comprising:
 13. Thenon-transitory computer-readable storage medium of claim 9, whereingenerating, based on the plurality of captured communications, thefiltered collection of captured communications further includes removingcaptured communications with destinations on an approved list ofdestinations, wherein the approved list of destinations indicatedestinations that are unlikely to be related to malware activity. 14.The non-transitory computer-readable storage medium of claim 9, whereinthe computer-executable instructions further configure the computersystem to perform operations comprising: identifying a quantity ofappearances of the new user-agent string in corresponding capturedcommunications among the first set of captured communications; anddetermining the quantity is below a predetermined threshold
 15. Acomputer system comprising: one or more computer readable storagedevices configured to store: a plurality of captured communicationsbetween an internal network and an external network; and a plurality ofuser-agent-related data items, wherein a data item of the plurality ofuser-agent-related data items comprises information associated with acomputing device; one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute computer executable instructions in order to causethe one or more hardware computer processors to: generate, based on theplurality of captured communications, a filtered collection of capturedcommunications by selecting captured communications that include auser-agent string; determine, based on the filtered collection ofcaptured communications, a first set of captured communicationsassociated with a test time period and a second set of capturedcommunications associated with a reference time period; identify a firstcaptured communication in the first set that is not included among thesecond set of captured communications, wherein the first capturedcommunication indicates a new user-agent string not previouslyassociated with the reference time period; and designate the newuser-agent string as a seed; and generate a data item cluster based onthe seed, wherein generating the data item cluster comprises: adding theseed to the data item cluster; and adding to the data item cluster oneor more user-agent-related data items from the plurality ofuser-agent-related data items determined to be associated with the seed.16. The computer system of claim 15, wherein the plurality ofuser-agent-related data items include at least one of: a user of aparticular computing device, an internal Internet Protocol address, anexternal Internet Protocol address, an external domain, an internalcomputing device, an external computing device, or a host-based event.17. The computer system of claim 15, wherein the one or more hardwarecomputer processors are further configured to cause presentation of thedata item cluster in a user interface of a client computing device. 18.The computer system of claim 15, wherein generating, based on theplurality of captured communications, the filtered collection ofcaptured communications further includes removing capturedcommunications with destinations on an approved list of destinations,wherein the approved list of destinations indicate destinations that areunlikely to be related to malware activity.
 19. The computer system ofclaim 15, wherein generating, based on the plurality of capturedcommunications, the filtered collection of captured communicationsfurther includes removing captured communications with respectiveuser-agent strings on an approved list of user-agent strings, whereinthe approved list of user-agent strings indicate communications that areunlikely to be related to malware activity.
 20. The computer system ofclaim 15, wherein generating, based on the plurality of capturedcommunications, the filtered collection of captured communicationsfurther includes removing captured communications associated with aparticular external computer system, wherein the particular externalcomputer system is unlikely to be related to malware activity.
 21. Thecomputer system of claim 15, wherein the one or more hardware computerprocessors are further configured to: identify a quantity of appearancesof the new user-agent string in corresponding captured communicationsamong the first set of captured communications; and determine thequantity is below a predetermined threshold.